Linux Device Drivers, 2nd Edition: Chapter 13: mmap and DMA
Linux Device Drivers, 2nd Edition
2nd Edition June 2001
0-, Order Number: 0081
586 pages, $39.95
Chapter 13
mmap and DMA
This chapter delves into the area of Linux memory management, with an
emphasis on techniques that are useful to the device driver writer.
The material in this chapter is somewhat advanced, and not everybody
will need a grasp of it. Nonetheless, many tasks can only be done
through digging more deeply into the memory
also provides an interesting look into how an important part of the
kernel works.
The material in this chapter is divided into three sections. The
first covers the implementation of the mmapsystem call, which allows the mapping of device memory directly into a
user process's address space. We then cover the kernel
kiobuf mechanism, which provides direct access to
user memory from kernel space. The kiobuf system
may be used to implement "raw I/O'' for certain kinds of devices.
The final section covers direct memory access (DMA) I/O operations,
which essentially provide peripherals with direct access to system
Of course, all of these techniques require an understanding of how
Linux memory management works, so we start with an overview of that
subsystem.
As with other parts of the kernel, both memory mapping and DMA have
seen a number of changes over the years. This section describes
the things a driver writer must take into account in order to write
portable code.
Changes to Memory Management
This chapter introduced the following symbols related to memory
handling. The list doesn't include the symbols introduced in the first
section, as that section is a huge list in itself and those symbols
are rarely useful to device drivers.
#include &linux/mm.h&
& 2001, O'Reilly & Associates, Inc.Tech and Media Labs
This site uses cookies to improve the user experience. OK
Java NIO vs. IO
When studying both the Java NIO and IO API's, a question quickly pops into mind:
When should I use IO and when should I use NIO?
In this text I will try to shed some light on the differences between Java NIO and IO,
their use cases, and how they affect the design of your code.
Main Differences Betwen Java NIO and IO
The table below summarizes the main differences between Java NIO and IO. I will get into
more detail about each difference in the sections following the table.
Stream oriented
Buffer oriented
Blocking IO
Non blocking IO
Stream Oriented vs. Buffer Oriented
The first big difference between Java NIO and IO is that IO is stream oriented,
where NIO is buffer oriented. So, what does that mean?
Java IO being stream oriented means that you read one or more bytes at a time, from a stream.
What you do with the read bytes is up to you. They are not cached anywhere. Furthermore,
you cannot move forth and back in the data in a stream. If you need to move forth and
back in the data read from a stream, you will need to cache it in a buffer first.
Java NIO's buffer oriented approach is slightly different. Data is read into a buffer from which
it is later processed. You can move forth and back in the buffer as you need to. This gives you
a bit more flexibility during processing. However, you also need to check if the buffer contains
all the data you need in order to fully process it. And, you need to make sure that when reading
more data into the buffer, you do not overwrite data in the buffer you have not yet processed.
Blocking vs. Non-blocking IO
Java IO's various streams are blocking. That means, that when a thread invokes a read()
or write(), that thread is blocked until there is some data to read, or the data is fully written.
The thread can do nothing else in the meantime.
Java NIO's non-blocking mode enables a thread to request reading data from a channel, and only get what is
currently available, or nothing at all, if no data is currently available. Rather than remain blocked until
data becomes available for reading, the thread can go on with something else.
The same is true for non-blocking writing. A thread can request that some data be written to a channel,
but not wait for it to be fully written. The thread can then go on and do something else in the mean time.
What threads spend their idle time on when not blocked in IO calls, is usually performing IO on other channels
in the meantime. That is, a single thread can now manage multiple channels of input and output.
Java NIO's selectors allow a single thread to monitor multiple channels of input. You can register
multiple channels with a selector, then use a single thread to "select" the channels that have input
available for processing, or select the channels that are ready for writing.
This selector mechanism makes it easy for a single thread to manage multiple channels.
How NIO and IO Influences Application Design
Whether you choose NIO or IO as your IO toolkit may impact the following aspects of your
application design:
The API calls to the NIO or IO classes.
The processing of data.
The number of thread used to process the data.
The API Calls
Of course the API calls when using NIO look different than when using IO. This is no surprise.
Rather than just read the data byte for byte from e.g. an InputStream, the data
must first be read into a buffer, and then be processed from there.
The Processing of Data
The processing of the data is also affected when using a pure NIO design, vs. an IO design.
In an IO design you read the data byte for byte from an InputStream or
a Reader. Imagine you were processing a stream of line based textual data.
For instance:
Name: Anna
This stream of text lines could be processed like this:
InputStream input = ... ; // get the InputStream from the client socket
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
String nameLine
= reader.readLine();
String ageLine
= reader.readLine();
String emailLine
= reader.readLine();
String phoneLine
= reader.readLine();
Notice how the processing state is determined by how far the program has executed. In other words,
once the first reader.readLine() method returns, you know for sure that a full line
of text has been read. The readLine() blocks until a full line is read, that's why.
You also know that this line contains the name. Similarly, when the second readLine()
call returns, you know that this line contains the age etc.
As you can see, the program progresses only when there is new data to read, and for each step you
know what that data is. Once the executing thread have progressed past reading a certain piece of data
in the code, the thread is not going backwards in the data (mostly not). This principle is also illustrated
in this diagram:
Java IO: Reading data from a blocking stream.
A NIO implementation would look different. Here is a simplified example:
ByteBuffer buffer = ByteBuffer.allocate(48);
int bytesRead = inChannel.read(buffer);
Notice the second line which reads bytes from the channel into the ByteBuffer.
When that method call returns you don't know if all the data you need is inside the buffer.
All you know is that the buffer contains some bytes. This makes processing somewhat harder.
Imagine if, after the first read(buffer) call, that all what was read into the
buffer was half a line. For instance, "Name: An". Can you process that data? Not really.
You need to wait until at leas a full line of data has been into the buffer, before it
makes sense to process any of the data at all.
So how do you know if the buffer contains enough data for it to make sense to be processed?
Well, you don't. The only way to find out, is to look at the data in the buffer. The result
is, that you may have to inspect the data in the buffer several times before you know if all
the data is inthere. This is both inefficient, and can become messy in terms of program design.
For instance:
ByteBuffer buffer = ByteBuffer.allocate(48);
int bytesRead = inChannel.read(buffer);
while(! bufferFull(bytesRead) ) {
bytesRead = inChannel.read(buffer);
The bufferFull() method has to keep track of how much data is read into the buffer,
and return either true or false, depending on whether the buffer is
full. In other words, if the buffer is ready for processing, it is considered full.
The bufferFull() method scans through the buffer, but must leave the buffer in
the same state as before the bufferFull() method was called. If not, the next
data read into the buffer might not be read in at the correct location. This is not impossible,
but it is yet another issue to watch out for.
If the buffer is full, it can be processed. If it is not full, you might be able to partially
process whatever data is there, if that makes sense in your particular case. In many cases it
The is-data-in-buffer-ready loop is illustrated in this diagram:
Java NIO: Reading data from a channel until all needed data is in buffer.
NIO allows you to manage multiple channels (network connections or files) using only a single (or few) threads,
but the cost is that parsing the data might be somewhat more complicated than when reading data from a blocking
If you need to manage thousands of open connections simultanously, which each only send a little data, for
instance a chat server, implementing the server in NIO is probably an advantage. Similarly, if you need
to keep a lot of open connections to other computers, e.g. in a P2P network, using a single thread to manage
all of your outbound connections might be an advantage. This one thread, multiple connections design is
illustrated in this diagram:
Java NIO: A single thread managing multiple connections.
If you have fewer connections with very high bandwidth, sending a lot of data at a time, perhaps a
classic IO server implementation might be the best fit. This diagram illustrates a classic IO server
Java IO: A classic IO server design - one connection handled by one thread.
Jakob Jenkov
Please enable JavaScript to view the114网址导航protobuf简介
一、什么是protobuf？
protobuf是以一种高效并且可扩展的格式去使数据结构化。google内部的RPC(远程过程调用，remote procedure
call)协议和文件的格式基本上用的都是protobuf。
二、为什么要使用protobuff？
我们将要使用一个简单的地址簿应用去阐释protobuf的应用过程，这个应用能够从一个文件当中读取和写入人们的通讯详细记录。每个人在地址簿当中有名字、ID、email和电话号码。&
& 也就是有这样一种数据关系结构:
&面对这样一种数据结构，我们如何序列化和检索数据呢？这里有几种方法去解决这个问题：
1、利用Java的序列化。这也是默认的方法。但是这样的话，如果我们需要和C++或者Python编写的应用去共享数据的话就不太好使了。
2、可以创造一种最高输出信道（ad-hoc）的方式去编码数据项使之成为一个单一的字符串类型。比如编码4
ints为"12:3:-23:67"。这是一种简单并且灵活的方式，尽管它需要一个额外的编码和解码过程，但是所需时间花费小。这种方式有利于非常简单的数据。
3、使数据序列化到XML。这个方式是非常吸引人的，因为XML可读性强并且有与多种语言相关的库。如果需要和其他的应用共享数据这不失为一个好的选择。但是，XML臭名昭著的就是需要的空间较大并且编码和解码过程都需要较大的性能消耗。另外没操作XML
DOM结点比在类中操纵一个属性复杂得多了。
&于是，相较而下，Protocol
buffers对于这类问题就显得更加的灵活、高效了。有了Protocol
buffers，我怕，我们可以写一个后缀为.proto的描述数据结构的文件。然后利用protobuf编译器自动生成一个类。这个类为自身的属性提供了get和set方法。最主要的是这种格式的数据很好进行扩展而且升级后的代码仍能读取老版本的数据格式。
所谓新版本或老版本可能就是在这个数据结构文件中新添加了一个属性，这样重新编译成的一个class文件，即使代码不进行更改也能对这个新的class文件，也就是数据结构进行读取。
三、定义协议格式（即后缀为.proto的文件）
可以在Eclipse中新建一个工程testprotobuf，在工程目录下，新建一个文件夹proto，在其下面新建一个文件msg.proto。
option java_package = "com.example.tutorial";
option java_outer_classname = "AddressBookProtos";
message Person {
& required string name = 1;
& required int32 id = 2;
& optional string email = 3;
& enum PhoneType {
& & MOBILE = 0;
& & HOME = 1;
& & WORK = 2;
& message PhoneNumber {
& & required string number =
& & optional PhoneType type
= 2 [default = HOME];
& repeated PhoneNumber phone = 4;
message AddressBook {
& repeated Person person = 1;
.proto文件是以一个包声明开头的，它的作用就是避免不同的工程项目之间的命名冲突。在Java当中，这个包名就充当java包来用除非我们像这里一样显性的定义java包"java_package"。反过来，即使定义了"java_package"，我们仍要定义"package"，虽然说现在java有了命名空间，在java应用中不会产生命名冲突，但是有可能在非Java应用中存在命名冲突的问题。
在声明"package"之后，我们看到两个option，这是Java的说明。java_package是说明java的包，所有的类都存在这个包下；java_outer_classname说明了类的名称，这个类下包含了我们定义的子类（比如：地址簿就是这个java_outer_classname指定类名的总类，其下又有人的子类）。如果我们没有定义这个java_outer_classname，那么它的默认名称和.proto的名称相同，只是是首字母是大写的，符合Java命名规范。
接下来就是信息的定义。一个信息仅仅只是一些列属性的集合。属性的标准类型有：bool、int32、float、double和string。我们能够添加更高级用message定义的结构类型。就像上面的例子，在信息AddressBook当中包含了Person的信息，而在信息Person当中又包含了PhoneNumber的信息。我们还可以定义enum的枚举类型。
在每个元素上的"=1","=2"的标记在二进制编码的属性当中被视为唯一的标签。标签1-15比较大的数要少一个字节的编码，所以我们为常用的或重复(reapted)使用的元素使用这样的标签以达到优化的目的；用16这样或更大数字的标签作为不常用的可选（optional）元素。每个重复（repeated）元素需要重复编码标签数字，所以重复属性作为一种很好的优化备选。
& &每个属性必须被下面的一个修饰词所注释：
·required：也就是必须提供这个属性的值，否则这个消息就被视为没有被初始化。如果试图绑定没有被初始化的消息就会抛出RuntimeException。而解析没有初始化的消息就会抛出IOException。
·optional：这个属性的属性值可以设定也可以不设定。如果属性值没有被设定，那么就使用默认值。我们可以自行的设置默认值。系统默认值对于数字类型的就为0，对于字符串就为空，布尔型的为false。对于嵌套的信息，默认为"default
instance"或者"prototype"，这样的信息没有属性集。
·repeated：这样的属性将会重复好几次。这个重复的属性值将会存储在协议缓存当中。我们可以把这个重复的属性当做一个动态的数组。
四、编译协议缓存
现在我们有了一个.proto文件，我们要将这个文件编译成class文件，于是我们就要用到编译器protoc。对于这个编译器的获取有两种方式：一种是下载源码自己手动对源码进行编译；另一种方式就是去google官网下载现成的编译器protoc.exe。假设现在编译器文件放在目录D:\complier\下。
& 在命令窗口当中：
D:\complier&protoc
--java_out=D:\EclipseWorkSpace\testProtoBuf\src msg.proto
命令解释：(1)protoc:编译器指令 (2)--java_out：以java文件的格式生成
(3)后面的地址为java文件存放的地址 (4)最后一个目录为需编译的.proto文件的目录
五、protobuf的API
经过编译，在D:\EclipseWorkSpace\testProtoBuf\src目录下就会生成一个AddressBookProtos.java的文件。现在让我们来看看生成的代码，看看编译器到底为我们生成了什么样的类和方法。如果我们打开AddressBookProtos.java，我们就能看到它定义了一个叫AddressBookProtos的类，其中嵌套了我们在addressbook.proto当中定义的message生成的类。每个类都有它自己的构造类(Builder
class)，我们可以用它来进行实例化。
每个消息和构造器都为消息的属性自动生成了存储器。消息只拥有get方法，构造器却有get和set方法。下面给出了人这个类的一些存储器方法。
// required string name = 1;
& & boolean hasName();
& & java.lang.String
getName();
// required int32 id = 2;
& & boolean hasId();
& & int getId();
// optional string email = 3;
& & boolean hasEmail();
& & java.lang.String
getEmail();
// repeated .test.Person.PhoneNumber phone = 4;
List&PhoneNumber&
getPhoneList();
& & PhoneNumber getPhone(int
getPhoneCount();
与此同时，我们再来看看Person.Builder产生的相同的get方法和额外的set方法。
// required string name = 1;
public boolean hasName();
public java.lang.String getName();
public Builder setName(）;
public Builder clearName();
// required int32 id = 2;
public boolean hasId();
&public int getId();
public Builder setId(int value) ;
public Builder clearId()；
&// optional string email = 3;
public boolean hasEmail() ;
public java.lang.String getEmail();
public Builder setEmail(java.lang.String value);
public Builder clearEmail();
// repeated .test.Person.PhoneNumber phone = 4;
public List&PhoneNumber&
getPhoneList();
public int getPhoneCount()；
public PhoneNumber getPhone(int index) ;
public Builder setPhone(int index, PhoneNumber value);
public Builder addPhone(PhoneNumber value);
public Builder clearPhone() ;
正如我们所看到的一样，它们拥有向JavaBean一样简单的set和get构造器。每个属性都有一个has的方法，如果返回true，说明这个属性已经赋予了属性值。最后每个属性还有一个clear方法，这个方法是那些没有赋值的属性返回到空的状态。
& 重复属性（Repeated
fields）拥有额外的方法-count方法能够根据返回具体的元素个数，add方法能够将新元素追加到元素列之后，addAll方法是整个容器都充满元素。
注意到这些方法都是采用的驼峰命名规则，即使在.proto文件中使用的是小写。这样的转换是通过protocol
buffer编译器自动完成的。所以我们应该在.proto的文件当中采用小写命名的规则。
六、构造方法(Builder)和信息(Message)
buffer产生的信息类是不可变的。一旦我们构造了一个信息对象，就不能对其进行修改，就像Java当中的String对象一样，都是用final关键字进行修饰的。我们首先要构造一个builder，然后才能设置我们想要的属性值，接着就是调用build()方法。
我们很容易看到每个构造器为对象设置属性后返回的仍是一个构造器，这样的话，我们就很容易实现链式编程。
& 下面就是我们创造的一个Person实例：
Person john=
& &Person.newBuider()
&.setId(1234)
&.setName("Jone Doe")
&.setEmail("");
&.addPhone(
&Person.PhoneNumber.newBuilder()
& &.setNumber("555-4321");
&.setType(Person.PhoneType.HOME)
).build();
七、解析与序列化
&每个协议缓存类都有写和读消息字节码的方法，它们包含有：
·byte[] toByteArray()：序列化消息并且返回一个包含原始字节的数组
·static Person parseFrom(byte[] data)：对给定的字节数组对消息进行解析
·void writeTo(OutputStream output)：序列化消息并且将它写给一个输出流
·static Person parseFrom(InputStream
input)：从一个输入流读取和解析消息
八、写消息
& 我们对这个地址簿应用第一件能想到的就是怎样往里面添加人物信息。
& 下面的程序就是根据用户的输入向文件当中输入人物信息。
package com.example.
import java.io.BufferedR
import java.io.FileInputS
import java.io.FileNotFoundE
import java.io.FileOutputS
import java.io.IOE
import java.io.InputStreamR
import java.io.PrintS
com.example.tutorial.AddressBookProtos.AddressB
import com.example.tutorial.AddressBookProtos.P
public class AddPerson {
public static void main(String[] args) throws Exception{
if(args.length!=1) {
System.err.println("Usage:AddPerson Address_book_file");
System.exit(-1);
AddressBook.Builder
addressBook=AddressBook.newBuilder();
addressBook.mergeFrom(new FileInputStream(args[0]));
} catch (FileNotFoundException e) {
& System.out.println(args[0]+":File not
found.Create a new File");
addressBook.addPerson(
PromptForAddress(new BufferedReader(new
InputStreamReader(System.in))
, System.out));
FileOutputStream output=new FileOutputStream(args[0]);
addressBook.build().writeTo(output);
output.close();
static Person PromptForAddress(BufferedReader stdin,
& & PrintStream stdout) throws
IOException {
Person.Builder person=Person.newBuilder();
stdout.print("Enter personId:");
person.setId(Integer.valueOf(stdin.readLine()));
stdout.print("Enter name:");
person.setName(stdin.readLine());
stdout.print("Enter eamil address(blank for nine):");
String email=stdin.readLine();
if(email.length()&0) {
person.setEmail(email);
while(true) {
stdout.print("Enter a phone number(or leaver blank to
finish):");
String number=stdin.readLine();
if(number.length()==0) {
Person.PhoneNumber.Builder phoneNumber=
Person.PhoneNumber.newBuilder().setNumber(number);
stdout.print("Is this a mobile,home,or work phone?");
String type=stdin.readLine();
if(type.equalsIgnoreCase("mobile")) {
phoneNumber.setType(Person.PhoneType.MOBILE);
}else if(type.equalsIgnoreCase("home")) {
phoneNumber.setType(Person.PhoneType.HOME);
}else if(type.equalsIgnoreCase("work")) {
phoneNumber.setType(Person.PhoneType.WORK);
stdout.println("Unknow phone type. Using default");
person.addPhone(phoneNumber);
return person.build();
九、读消息
& 下面的程序就是从地址簿当中读取人物的信息。
package com.example.
import java.io.FileInputS
com.example.tutorial.AddressBookProtos.AddressB
import com.example.tutorial.AddressBookProtos.P
public class ListPeople {
public static void main(String[] args) throws Exception
if(args.length!=1) {
System.err.println("Usage:ListPeople
Address_book_file");
System.exit(-1);
AddressBook addressBook=
AddressBook.parseFrom(new FileInputStream(args[0]));
print(addressBook);
static void print(AddressBook addressBook) {
for(Person person:addressBook.getPersonList()) {
System.out.println("Person Id:"+person.getId());
System.out.println("Name:"+person.getName());
if(person.hasEmail()) {
System.out.println("E-mail address:"+person.getEmail());
for(Person.PhoneNumber phoneNumber:person.getPhoneList())
switch(phoneNumber.getType()) {
case MOBILE:
System.out.println(" Mobile Phone #:");
case HOME:
System.out.println(" Home Phone #:");
case WORK:
System.out.println(" Work Phone #:");
System.out.println(phoneNumber.getNumber());
已投稿到：
以上网友发言只代表其个人观点，不代表新浪网的观点或立场。}