Saturday, October 31, 2009

Introduction















































Introduction



This book is about using cryptography with Java.


Depending on who you are, you may think of cryptography
as a great menace or as a very useful tool. The truth is that in some
ways it is neither, in other ways it is both. It's neither because, if
you choose, you could easily reduce it to an interesting mathematical
game, with no application. It's both because it most definitely gets
applied, not always well, and not always for purposes that everyone
agrees with. Whichever side of the fence you are on, the one thing
everyone agrees with is that the politics surrounding cryptography and
access to the technology that allows you to use it have been intense.


Java, on the other hand, is simply a programming
language. It arrived on the popular scene in 1995 and has become very
popular as a language for writing applications involving the Internet,
electronic commerce, or a combination of the two. Other than the odd,
often "religious," issue between programmers or companies, the language
itself has carried none of the political problems that have accompanied
cryptography. However, a language by itself won't allow you to develop
secure Internet applications, so it quickly became apparent that it
would be necessary to introduce into Java APIs that allowed people to
make use of cryptography. When this happened the politics arrived and,
for some, using Java suddenly got intense as well. Finally, the
politics subsided and we arrived where we are now, with a rich set of
APIs that allow developers to use cryptography effectively in
application development—providing they know how.


This brings me to why this book was written. People
will still wax lyrically about key sizes and PKI (Public Key
Infrastructure), but what does it all mean, and what does it mean when
you are using Java? The Java APIs afford you a great deal of
flexibility, and although this will allow you to implement an
application using cryptography at a fundamental level, it will also
allow you to tie yourself into some terrible knots. You can avoid this
if you understand a few principals about the way the APIs are put
together. Furthermore, by understanding the relationships between the
high-level APIs and the more fundamental ones, debugging becomes easier
and you can recognize when it is not necessary to build things from
scratch, as, in some cases, the hard work has already been done. In
short, with the right understanding, you can save yourself a lot of
work. This book has been written with the aim of providing that
understanding.




Who This Book Is For


This book is written for people who are Java
developers and are trying to make use of cryptography in their
applications and for people who simply want to understand what's going
on when cryptography is being used in Java applications. It does assume
you are familiar with the Java language, but it does not assume you
have any familiarity with any of the APIs it discusses, such as the
JCA, JCE, the Bouncy Castle APIs, and JavaMail.


If you are already very familiar with the JCE and
the JCA, you might want to skim the first four chapters quickly and
start reading thoroughly from Chapter 5
onward; otherwise, I would recommend you start at the beginning. If you
do skim the first four chapters, you should pay attention to the
development of the utilities class that is added at the start of most
chapters. The reason is that the utilities class used in Chapter 5 and onward builds on the work done in the first four chapters.








































Chapter 1. Introduction to Agile Java Development











1. Introduction to Agile Java Development










WHEN JAVA DEVELOPMENT KIT (JDK) v1.0 was released in January 1996, it was a fairly straightforward application programming interface (API). Over the years, Java has matured into a full-blown platform. From JDK 1.0 to JDK 1.5, we have been introduced to many new features, such as the Java Collections Framework, logging API, auto-boxing, generics, and more. Although most of these are useful, Java has also become more complex, especially after the advent of the Java Platform Enterprise Edition (JEE). JEE introduced such concepts as Enterprise JavaBeans (EJB), which sought to simplify vendor-portable, enterprise-level distributed computing, but instead, it introduced unnecessary complexities for 80% of the applications out there. Nowadays, it is not uncommon for many people to think of Java/JEE as being a big and heavy technology. Well, for starters, this couldn't be further from the truth, and second, let's see if we can change this perspective in this book.


In the past few years, many open source frameworks have sprung up to solve some of the problems created by JEE. This book covers some of these open source frameworks (for example, Spring and Hibernate) as well as open source tools (such as Ant and Eclipse), which provide a comprehensive, effective, and elegant solution that can either be viewed as complementary or as a complete alternative to JEE, depending on how you apply these technologies for your specific needs. In addition, nimble software development processes such as Extreme Programming (XP) and Agile Model Driven Development (AMDD) can assist in accelerating the project delivery.


Software development is about people, processes, and technology (and probably in that order of priority). The people are the stakeholders, the customer we build software for. In this book, I will cover the process and technology parts. You will learn how to leverage these tools and technologies to rapidly develop end-to-end applications using Java, from the client tier to the data tier, and more. Along the way, you should see many of the benefits resulting from using these tools and technologiesfor example, simplicity and speed of development.


Before we begin, if you have not read the preface, I would recommend at least glancing through it because it provides some foundation for the goals of this book and the way it is organized, and includes some reasons why I wrote this book.















Lab 8.1 Exercise Answers



[ Team LiB ]





Lab 8.1 Exercise Answers


This section gives you some suggested answers to the questions in Lab 8.1, with discussion related to how those answers resulted. The most important thing to realize is whether your answer works. You should figure out the implications of the answers here and what the effects are from any different answers you may come up with.


8.1.1 Answers


a)

What output was printed on the screen?

A1:

Answer: Your output should look like the following:



v_counter = 1
v_counter = 2
v_counter = 3
v_counter = 4
v_counter = 5
Done...

PL/SQL procedure successfully completed.



Every time the loop is run, the statements in the body of the loop are executed. In this script, the value of v_counter is incremented by 1 and displayed on the screen. The EXIT condition is evaluated for each value of v_counter. Once the value of v_counter increases to 5, the loop is terminated. For the first iteration of the loop, the value of v_counter is equal to 1, and it is displayed on the screen, and so forth. After the loop has terminated, "Done..." is displayed on the screen.


b)

How many times was the loop executed?

A2:

Answer: The loop was executed five times.



Once the value of v_counter increases to 5, the IF statement





IF v_counter = 5 THEN
EXIT;
END IF;

evaluates to TRUE, and the loop is terminated.


The loop counter tracks the number of times the loop is executed. You will notice that in this exercise, the maximum value of v_counter is equal to the number of times the loop is iterated.


c)

What is the EXIT condition for this loop?

A3:

Answer: The EXIT condition for this loop is v_counter = 5.



The EXIT condition is used as a part of an IF statement. The IF statement evaluates the EXIT condition to TRUE or FALSE, based on the current value of v_counter.


d)

How many times will the value of the variable v_counter be displayed if the DBMS_OUTPUT.PUT_LINE statement is used after the END IF statement?

A4:

Answer: The value of v_counter will be displayed four times.



LOOP
v_counter := v_counter + 1;
IF v_counter = 5 THEN
EXIT;
END IF;
DBMS_OUTPUT.PUT_LINE ('v_counter = '||v_counter);
END LOOP;



Assume that the loop has iterated four times already. Then the value of v_counter is incremented by 1, so v_counter is equal to 5. Next, the IF statement evaluates the EXIT condition. The EXIT condition yields TRUE, and the loop is terminated. The DBMS_OUTPUT.PUT_LINE statement is not executed for the fifth iteration of the loop because control is passed to the next executable statement after the END LOOP statement. Thus, only four values of v_counter are displayed on the screen.


e)

Why does the number of times the loop counter value is displayed on the screen differ when the DBMS_OUTPUT.PUT_ LINE statement is placed after the END IF statement?

A5:

Answer: When the DBMS_OUTPUT.PUT_LINE statement is placed before the IF statement, the value of v_counter is displayed on the screen first. Then it is evaluated by the IF statement. The fifth iteration of the loop "v_counter = 5" is displayed first, then the EXIT condition yields TRUE and the loop is terminated.

When the DBMS_OUTPUT.PUT_LINE statement is placed after the END IF statement, the EXIT condition is evaluated prior to the execution of the DBMS_OUTPUT.PUT_ LINE statement. Thus, for the fifth iteration of the loop, the EXIT condition evaluates to TRUE before the value of v_counter is displayed on the screen by the DBMS_OUTPUT.PUT_LINE statement.

f)

Rewrite this script using the EXIT WHEN condition instead of the EXIT condition, so that it produces the same result.

A6:

Answer: Your script should look similar to the following script. Changes are shown in bold letters.



-- ch08_1b.sql, version 2.0
SET SERVEROUTPUT ON
DECLARE
v_counter BINARY_INTEGER := 0;
BEGIN
LOOP
-- increment loop counter by one
v_counter := v_counter + 1;
DBMS_OUTPUT.PUT_LINE ('v_counter = '||v_counter);

-- if EXIT WHEN condition yields TRUE exit the loop
EXIT WHEN v_counter = 5;
END LOOP;

-- control resumes here
DBMS_OUTPUT.PUT_LINE ('Done...');
END;



Notice that the IF statement has been replaced by the EXIT WHEN statement. The rest of the statements in the body of the loop do not need to be changed.


8.1.2 Answers


a)

How many sections will be added for the specified course number?

A1:

Answer: Four sections were added for the given course number.

b)

How many times will the loop be executed if the course number is not valid?

A2:

Answer: The loop will be executed one time.



If the course number is not valid, the INSERT statement





INSERT INTO section
(section_id, course_no, section_no, instructor_id,
created_date, created_by, modified_date, modified_by)
VALUES
(section_id_seq.nextval, v_course, v_sec_num,
v_instructor_id, SYSDATE, USER, SYSDATE, USER);

will cause an exception to be raised. As soon as an exception is raised, control is passed out of the loop to the exception handler. Therefore, if the course number is not valid, the loop will be executed only once.


c)

How would you change this script to add 10 sections for the specified course number?

A3:

Answer: Your script should look similar to the following script. Changes are shown in bold letters.



-- ch08_2b.sql, version 2.0
DECLARE
v_course course.course_no%type := 430;
v_instructor_id instructor.instructor_id%type := 102;
v_sec_num section.section_no%type := 0;
BEGIN
LOOP
-- increment section number by one
v_sec_num := v_sec_num + 1;
INSERT INTO section
(section_id, course_no, section_no,
instructor_id, created_date, created_by,
modified_date, modified_by)
VALUES
(section_id_seq.nextval, v_course, v_sec_num,
v_instructor_id, SYSDATE, USER, SYSDATE,
USER);

-- if number of sections added is ten exit the loop
EXIT WHEN v_sec_num = 10;
END LOOP;

-- control resumes here
COMMIT;
EXCEPTION
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE ('An error has occurred');
END;



In order to add 10 sections for the given course number, the test value of v_sec_num in the EXIT condition is changed to 10.


Note that before you execute this version of the script you need to delete records from the SECTION table that were added when you executed the original example. If you did not run the original script, you do not need to delete records from the SECTION table.


The SECTION table has a unique constraint defined on the COURSE_NO and SECTION_NO columns. In other words, the combination of course and section numbers allows you to uniquely identify each row of the table. When the original script is executed, it creates four records in the SECTION table for course number 430, section numbers 1, 2, 3, and 4. When the new version of this script is executed, the unique constraint defined on the SECTION table is violated because there already are records corresponding to course number 430 and section numbers 1, 2, 3, and 4. Therefore, these rows must be deleted from the SECTION table as follows:





DELETE FROM section
WHERE course_no = 430
AND section_no <= 4;

Once these records are deleted from the SECTION table, you can execute the new version of the script.


d)

How would you change the script to add only even-numbered sections (maximum section number is 10) for the specified course number?

A4:

Answer: Your script should look similar to the following script. Changes are shown in bold letters. In order to run this script, you will need to delete records from the SECTION table that were added by the previous version. With each iteration of the loop, the value of v_sec_num should be incremented by two, as shown:



-- ch08_2c.sql, version 3.0
SET SERVEROUTPUT ON
DECLARE
v_course course.course_no%type := 430;
v_instructor_id instructor.instructor_id%type := 102;
v_sec_num section.section_no%type := 0;
BEGIN
LOOP
-- increment section number by two
v_sec_num := v_sec_num + 2;
INSERT INTO section
(section_id, course_no, section_no,
instructor_id, created_date, created_by,
modified_date, modified_by)
VALUES
(section_id_seq.nextval, v_course, v_sec_num,
v_instructor_id, SYSDATE, USER, SYSDATE,
USER);

-- if number of sections added is ten exit the loop
EXIT WHEN v_sec_num = 10;
END LOOP;

-- control resumes here
COMMIT;
EXCEPTION
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE ('An error has occurred');
END;
e)

How many times will the loop be executed in this case?

A5:

Answer: The loop is executed five times when even-numbered sections are added for the given course number.







    [ Team LiB ]



    Chapter 8: Behavioral Patterns




















    Chapter 8 -
    Behavioral Patterns
    Patterns in Java, Volume 1: A Catalog of Reusable Design Patterns Illustrated with UML, Second Edition
    by Mark Grand
    John Wiley & Sons � 2002


























    Chapter 8: Behavioral Patterns



    The patterns in this chapter are used to organize, manage, and combine behavior.




    Chain of Responsibility



    This pattern was previously described in [GoF95].




    Synopsis


    The Chain of Responsibility pattern allows an object to send a command without knowing what object or objects will receive it. It accomplishes this by passing the command to a chain of objects that is typically part of a larger structure. Each object in the chain may handle the command, pass the command on to the next object in the chain, or do both.






    Context


    Suppose that you are writing software to monitor a security system. Physically, the security system consists of sensing devices (motion detectors, smoke detectors, etc.) that transmit status information to a computer. The computer’s job is to log all status information, maintain a display showing current status information, and transmit alarms in the event of an emergency.


    One of the goals for the monitoring software is that it should be highly scalable. It should be able to work for a small retail store, an office building, a warehouse, or a multi-building complex. That goal has implications for the way that you design the monitoring software.


    To keep things simple, your monitoring program should instantiate an object for every sensor it is to monitor. This provides a simple way to model each sensor’s state. To ensure scalability, an object responsible for an individual sensor should not assume anything about its environment, except that it is at the bottom level of a hierarchical organization.


    The organization will include objects corresponding to real-world things such as rooms, areas, floors, and buildings. Directly modeling the real world provides a straightforward way to display the status of different parts of buildings. It also allows the interpretation of a sensor’s state to be based on its environment. For example, if the temperature of a closed room exceeds 180°F, then you may want the fire sprinklers in just that room to turn on. If the temperature in an open area of a warehouse exceeds 150°F, you may want to turn on the fire sprinklers over that area and the adjacent areas. On the other hand, if the temperature in a freezer exceeds 30°F, you may want to sound an alarm to let people know that that freezer is getting too warm.


    In all these cases, the object that models a sensor does not decide what to do with the state of the sensor. Instead, it delegates that decision to an object at a higher level of the hierarchy that has more contextual knowledge. Such objects either decide what to do about a notification or pass it on to the object that is organizationally above it.



    Figure 8.1 shows an example of objects organized in this hierarchical way.






    Figure 8.1: Physical security object organization

    For example, when a TemperatureSensor object contained in an area of a warehouse receives a notification of the current temperature from the physical sensor, it passes that notification to the Area object that contains it. Rather than decide the significance of the temperature, it passes the notification to the Warehouse object that contains the Area object. The Warehouse object determines the meaning of the temperature. If the temperature is above 150°F, the Warehouse object decides that there is a fire. It turns on the sprinklers in the area that notified it and the surrounding areas. The Warehouse object does not pass on the temperature notification.







    Forces




















    J



    You want an object to be able to send a command to another object without specifying the receiver. The sending object does not care which object handles the command, only that an object will receive the command and handle it.




    J



    You want the receivers of a command to be able to handle the command without having to know anything about the object that sent the command.




    J



    More than one object may be able to receive and handle a command, so you need a way to prioritize among the receivers without the sending object knowing anything about them.




    J



    The objects you may want to potentially handle commands are organized into a structure that can serve to prioritize among the potential handlers of a command.








    Solution



    Figure 8.2 presents a class diagram that shows the organization of the Chain of Responsibility pattern. Following are explanations of the roles these classes play in the Chain of Responsibility pattern:






    Figure 8.2: Chain of Responsibility pattern.


    CommandSender.  Instances of a class in this role send commands to the first object in a chain of objects that may handle the command. They send a command by calling the first CommandHandlerIF object’s postCommand method.



    CommandHandlerIF.  All objects in a chain of objects that may handle a command must implement an interface in this role. It defines two methods.




    1. It defines a handleCommand method to handle whatever commands an implementing class is expected to handle. The handleCommand method returns true if it handled a command or false if it did not.




    2. It defines a postCommand method that calls the handleCommand method. If the handleCommand method returns false and there is a next object in the chain, it calls the object’s postCommand method. If the handleCommand method returns true, it means there is no need to pass the command on to the next object in the chain.



    AbstractCommandHandler.  Classes in this role are abstract classes that implement the postCommand method. The purpose of this is to provide the convenience of a common implementation of postCommand for classes in the ConcreteCommandHandler role. It is very unusual for classes to want anything other than the default logic for the postCommand method. Classes in the CommandSender role should refer to objects in a chain of responsibility only through the CommandHandlerIF interface and not as instances of the AbstractCommandHandler class. The AbstractCommandHandler class is an implementation detail. Though unusual, it is possible to have classes that implement the CommandHandlerIF interface that are not subclasses of the AbstractCommandHandler class.



    ConcreteCommandHandler1, ConcreteCommandHandler2, and so on.  Instances of classes in this role are objects in a chain of objects that can handle commands.


    Typically, CommandHandler objects are part of a larger structure. This is the case in the example shown in Figure 8.1.






    Implementation


    In many cases, the objects that constitute a chain of responsibility are part of a larger structure, and the chain of responsibility is formed through some links of that larger structure. When links to form a chain of responsibility do not already exist, you must add instance variables and access methods to the classes to create links that form a chain of responsibility.


    A decision to make, whenever implementing the Chain of Responsi bil ity pattern, is how you will pass commands to and through the chain of objects. There are two basic ways to do it. One way is to encapsulate each kind of command in a single object that can be passed to a single postCommand method. The other way is to have as many different types of postCommand and handleCommand methods as there are different types of information associated with commands.


    Passing commands in a single object is often the better choice. It incurs the cost of object creation but minimizes the cost of passing parameters to the methods of the next object in the chain. That minimizes the cost of propagating a command through a chain of objects. Passing commands in a single object usually results in less code.


    On the other hand, passing the information that constitutes a command through separate parameters saves the cost of object creation at the cost of additional parameter passing. If you know that the chain of objects will be short, passing a command as multiple parameters can be the better choice.






    Consequences




















    J



    The Chain of Responsibility pattern reduces coupling between the object that sends a command and the object that handles the command. The sender of a command does not need to know what object will actually handle the command. It merely needs to be able to send the command to the object that is at the head of the chain of responsibility.




    J



    The Chain of Responsibility pattern allows flexibility in deciding how to handle commands. Decisions about which object will handle a command can be varied by changing which objects are in the chain of responsibility or changing the order of the objects in the chain of responsibility.




    l



    The Chain of Responsibility pattern does not guarantee that every command will be handled. Commands that are not handled are ignored.




    J



    If the number of objects in a chain becomes large, there can be efficiency concerns about the amount of time that it takes a command to propagate through the chain. A high percentage of commands that are not handled exacerbates the problem because commands that are not handled are propagated through the full length of the chain.








    Java API Usage



    Version 1.0 of Java used the Chain of Command pattern to handle user- interface events. That event-handling scheme used a user interface’s container hierarchy as a chain of responsibility. When an event was posted to a button or other GUI component, it would either handle the event or post it to its container. Though it was usable, there were enough problems that the creators of Java took the drastic step of changing Java’s event model. The two most serious problems related to efficiency and flexibility are as follows:














    l



    Some platforms generate many events that most GUIs do not handle or have any interest in. One such event is MOUSE_MOVE. It may be generated every time a mouse moves just one pixel. Some programs that were built using the original event model visibly slowed down whenever there was rapid mouse movement because they spent so much time passing MOUSE_MOVE events that were never handled through the container hierarchy.




    l



    The Chain of Responsibility pattern assumes that all the objects that can handle a command are instances of a common superclass or implement a common interface. This limits a program to posting commands to instances of that common superclass or interface. Java’s original event model required that every object that could handle an event was an instance of the common superclass Component. This meant that it was impossible to deliver events directly to non-GUI objects, since only GUI objects are instances of Component.




    This second problem is actually an advantage for some applications. The Chain of Responsibility pattern makes it less convenient to deliver commands to objects that are not part of the chain of handlers. For some applications, delivering a command to an object outside the chain of handlers is most likely a bug. The physical security example discussed under the Context heading is an example of such an application. For applications such as this, using the Chain of Responsibility pattern will result in fewer opportunities to introduce bugs into the application than will be using the delegation event model.


    Another advantage that the Chain of Responsibility has over the delegation event model is that it allows you to explicitly control the order in which commands are delivered to handlers.






    Code Example


    Continuing the physical security example, Figure 8.3 shows the classes used in the physical security example.








    In Figure 8.3, the classes that extend the Sensor class call the notify method they inherit from it to report a measurement to the object that is responsible for handling its measurements. Classes that extend the AbstractSecurityZone class are responsible for handling measurements from the appropriate kind of Sensor object.






    Figure 8.3: Physical security classes.


    The following is some code for the classes shown in Figure 8.3. First, here is code for the TemperatureSensor class. Notice that the TemperatureSensor class does nothing with a reading from a temperature sensor but pass it on.




    class TemperatureSensor extends Sensor {
        private SecurityZone zone;
    ...
    /**
    * When the temperature sensor associated with this object
    * observes a different temperature this method is called.
    */
    void notify(int measurement) {
    zone.notify(measurement, this);
    } // notify(int)
    } // class TemperatureSensor


    All of the classes that model security zones implement that SecurityZoneIF interface:




    public interface SecurityZoneIF {
    /**
    * This method is called to notify this security zone of a
    * change in a sensor measurement.
    */
        public void notify(int measurement, Sensor source) ;

    /**
    * This method is called by a child zone to report a fire.
    */
        public void fireAlarm(SecurityZone zone) ;
    }  // interface SecurityZoneIF


    Here is the code for the SecurityZone class, which is the superclass of all of the classes that form the chains of responsibility in this example:



    abstract class SecurityZone implements SecurityZoneIF { 
        private SecurityZone parent;
    ...
    /**
    * Return this object's parent zone.
    */
        SecurityZone getParent() {
            return parent;
    } // getParent()

    /**
    * Call this method to notify this zone of a new sensor
    * measurement.
    */
        public void notify(int measurement, Sensor sensor) {
            if (!handleNotification(measurement, sensor)
                  && parent != null) {
                parent.notify(measurement, sensor);
    } // if
    } // notify(int, Sensor)

    /**
    * This method is called by the notify method so that
    * this object can have a chance to handle measurements.
    */
        protected
        abstract boolean handleNotification(int measurement,
                                            Sensor sensor);
    /**
    * This method is called by a child zone to report a fire.
    * It is expected that the child zone has turned on
    * sprinklers or taken other measures to control the fire
    * within the child zone. The purpose of this method is to
    * be overridden by subclasses so it can take any
    * necessary actions outside of the child zone.
    */
        public void fireAlarm(SecurityZone zone) {
    // Turn on sprinklers
    ...
            if (parent != null)
              parent.fireAlarm(zone);
    } // fireAlarm(SecurityZone)
    } // class SecurityZone


    Here are the subclasses of SecurityZone that were discussed under the Context heading:




    class Area extends SecurityZone {
    ...
    /**
    * This method is called by the notify method so that this
    * object can have a chance to handle measurements.
    */
        boolean handleNotification(int measurement,Sensor sensor){
            if (sensor instanceof TemperatureSensor) {
                if (measurement > 150) {
                    fireAlarm(this);
                    return true;
                }  // if
    } // if
    ...
            return false;
    } // handleNotification(int, Sensor)
    } // class Area

    class Warehouse extends SecurityZone {
    ...
    /**
    * This method is called by the notify method so this
    * object can have a chance to handle measurements.
    */
        protected
        boolean handleNotification(int measurement,Sensor sensor){
    ...
            return false;
    } // handleNotification(int, Sensor)

    public void fireAlarm(SecurityZone zone) {
            if (zone instanceof Area) {
    // Turn on sprinklers in surrounding areas
    ...
    // Don't call super.fireAlarm because that will
    // turn on sprinklers for the whole warehouse.
                if (getParent() != null)
                  getParent().fireAlarm(zone);
                return;
    } // if
    ...
            super.fireAlarm(zone);
    } // fireAlarm(SecurityZone)
    }  // class Warehouse






    Related Patterns



    Composite.  When the chain of objects used by the Chain of Responsibility pattern is part of a larger structure, the larger structure is usually built using the Composite pattern.



    Command.  The Chain of Responsibility pattern makes the particular object that executes a command indefinite. The Command pattern makes the object that executes a command explicit and specific.



    Template Method.  When the objects that make up a chain of responsibility are part of a larger organization built using the Composite pattern, the Template Method pattern is often used to organize the behavior of individual objects.


















    Section 21.5. Tuning DML (INSERT, UPDATE, DELETE)










    21.5. Tuning DML (INSERT, UPDATE, DELETE)









    The first principle for optimizing UPDATE, DELETE, and INSERT statements is to optimize any WHERE clause conditions used to find the rows to be manipulated or inserted. The DELETE and UPDATE statements may contain WHERE clauses, and the INSERT statement may contain SQL that defines the data to be inserted. Ensure that these WHERE clauses are efficientperhaps by creating appropriate concatenated indexes
    .


    The second principle for optimizing DML performance is to avoid creating too many indexes. Whenever a row is inserted or deleted, updates must occur to every index that exists against the table. These indexes exist to improve query performance, but bear in mind that each index also results in overhead
    when the row is created or deleted. For updates, only the indexes that reference the specific columns being modified need to be updated.



    21.5.1. Batching Inserts


    The MySQL language allows more than one row to be inserted in a single INSERT operation. For instance, the statement in Example 21-21 inserts five rows into the clickstream_log table in a single call.


    Example 21-21. Batch INSERT statement




    INSERT INTO clickstream_log (url,timestamp,source_ip)
    values
    ('http://dev.mysql.com/downloads/mysql/5.0.html',
    '2005-02-10 11:46:23','192.168.34.87') ,
    ('http://dev.mysql.com/downloads/mysql/4.1.html',
    '2005-02-10 11:46:24','192.168.35.78'),
    ('http://dev.mysql.com',
    '2005-02-10 11:46:24','192.168.35.90'),
    ('http://www.mysql.com/bugs',
    '2005-02-10 11:46:25','192.168.36.07'),
    ('http://dev.mysql.com/downloads/mysql/5.1.html',
    '2005-02-10 11:46:25','192.168.36.12')




    Batching INSERT operations in this way can radically improve performance. Figure 21-10 shows how the time taken to insert 10,000 rows into the table decreases as we increase the number of rows included within each INSERT statement. Inserting one row at a time, it took about 384 seconds to insert the rows. When inserting 100 rows at a time, we were able to add the same number of rows in only 7 seconds.



    Figure 21-10. Performance improvement from multirow inserts



    Whenever possible, use MySQL's multirow insert feature to speed up the bulk loading of records.





    21.5.2. Optimizing DML by Reducing Commit Frequency




    If we are using a transactional storage enginefor instance, if our tables are using the InnoDB enginewe should make sure that we are committing changes to the database only when necessary. Excessive commits will degrade performance.


    By default, MySQL will issue an implicit commit after every SQL statement. When a commit occurs, a storage engine like InnoDB will write a record to its transaction log on disk to ensure that the transaction is persistent (i.e., to ensure that the transaction will not be lost if MySQL or our program crashes). These transaction log writes involve a physical I/O to the disk and therefore always add to our response time.


    We can prevent this automatic commit behavior by issuing the SET AUTOCOMMIT=0 statement and/or by issuing a START TRANSACTION statement before issuing our statements. We can then issue a COMMIT statement at regular intervals, reducing the number of writes to the transaction log that will be required. (Note, though, that MySQL will occasionally write to the transaction log anyway when memory buffers require flushing.)


    Usually, the frequency with which we commit is driven by our application logic rather than by performance. For instance, if a user clicks a Save button in our application, he is going to expect that the information will be permanently saved to the database, and so we will be required to issue a COMMIT as a result. However, in batch applications, we can often choose to commit at relatively infrequent intervals. Reducing the commit frequency can have a huge effect on DML performance.


    In Figure 21-11, we see how reducing the commit frequency affected the time taken to insert 10,000 rows into the database. At the default settings, it took about 850 seconds (about 14 minutes) to insert the 10,000 rows. If we commit only after every 100 rows have been inserted, the time taken is reduced to only 8 seconds.


    In these tests, the InnoDB transaction log was on the same disk as the InnoDB tablespace files, which magnified the degradation caused by transaction log writes. Moving the transaction log to a dedicated disk can reducealthough not eliminatethe transaction log overhead.



    Figure 21-11. How commit frequency affects DML performance



    When you are using a transactional storage engine (such as InnoDB) in situations where your application logic permits (batch applications, for instance), reducing the frequency at which you commit work can massively improve the performance of INSERTs, UPDATEs, and DELETEs.



    We looked at how you can manipulate commit frequency in stored programs in Chapter 8.




    21.5.3. Triggers and DML Performance


    Because trigger code will be invoked for every row affected by the relevant DML operation, poorly performing triggers can have a very significant effect on DML performance
    . If our DML performance is a concern and there are triggers on the tables involved, we may want to determine the overhead of our triggers by measuring
    performance with and without the triggers.


    We provide some more advice on trigger tuning in Chapter 22.













    Recipe 10.12. Evaluating Code in an Earlier Context










    Recipe 10.12. Evaluating Code in an Earlier Context





    Problem


    You've written a method that evaluates a string as Ruby code. But whenever anyone calls the method, the objects referenced by your string go out of scope. Your string can't be evaluated within a method.


    For instance, here's a method that takes a variable name and tries to print out the value of the variable.



    def broken_print_variable(var_name)
    eval %{puts "The value of #{var_name} is " + #{var_name}.to_s}
    end



    The eval code only works when it's run in the same context as the variable definition. It doesn't work as a method, because your local variables go out of scope when you call a method.



    tin_snips = 5

    broken_print_variable('tin_snips')
    # NameError: undefined local variable or method 'tin_snips' for main:Object

    var_name = 'tin_snips'
    eval %{puts "The value of #{var_name} is " + #{var_name}.to_s}
    # The value of tin_snips is 5





    Solution


    The eval method can execute a
    string of Ruby code as though you had written in some other part of your application. This magic is made possible by Binding objects. You can get a Binding at any time by calling Kernel#binding, and pass it in to eval to recreate your original environment where it wouldn't otherwise be available. Here's a version of the above method that takes a Binding:



    def print_variable(var_name, binding)
    eval %{puts "The value of #{var_name} is " + #{var_name}.to_s}, binding
    end

    vice_grips = 10
    print_variable('vice_grips', binding)
    # The value of vice_grips is 10





    Discussion


    A Binding object is a bookmark of the Ruby interpreter's state. It tracks the values of any local variables you have defined, whether you are inside a class or method definition, and so on.


    Once you have a Binding object, you can pass it into eval to run code in the same context as when you created the Binding. All the local variables you had back then will be available. If you called Kernel#binding within a class definition, you'll also be able to define new methods of that class, and set class and instance variables.


    Since a Binding object contains references to all the objects that were in scope when it was created, those objects can't be garbage-collected until both they and the Binding object have gone out of scope.




    See Also


    • This trick is used in several places throughout this book; see, for example, Recipe 1.3, "Substituting Variables into an Existing String," and Recipe 10.9, "Automatically Initializing Instance Variables"













    Encryption Examples




















    Encryption Examples



    The purpose of this section is to give an overview of how encryption can be performed with different data. You’ll see a few examples of encrypting data with various algorithms. To start, a simple encryption of character data is presented. The output will be viewed both as a string and as a RAW. The decryption is done to verify that you can successfully recover the original data:



    sec_mgr@KNOX10g> DECLARE    
    2 l_plaintext VARCHAR2 (12) := 'This is data';
    3 l_key VARCHAR2 (20) := 'This is the Key';
    4 l_encrypted_text VARCHAR2 (24);
    5 l_encrypted_raw RAW (24);
    6 l_decrypted_text VARCHAR2 (`20);
    7 BEGIN
    8 l_encrypted_raw := data_crypto.encrypt_char (l_plaintext, l_key);
    9 l_encrypted_text := data_crypto.encrypt (l_plaintext, l_key);
    10 l_decrypted_text := data_crypto.decrypt (l_encrypted_text, l_key);
    11 DBMS_OUTPUT.put_line ('PlainText: ' || l_plaintext);
    12 DBMS_OUTPUT.put_line ('Key: ' || l_key);
    13 DBMS_OUTPUT.put_line ('Encrypted RAW: ' || l_encrypted_raw);
    14 DBMS_OUTPUT.put_line ('Encrypted string: ' || l_encrypted_text);
    15 DBMS_OUTPUT.put_line ('Decrypted: ' || l_decrypted_text);
    16 END;
    17 /
    PlainText: This is data
    Key: This is the Key
    Encrypted RAW: A84513EC540BF7E87BCA3AF29A0CAD34
    Encrypted string: e ^ ' = 8. = T^¯óRí
    Decrypted: This is data


    Invoking encryption is done very easily. The challenge remains as to how to store encrypted data and how to mange the encryption keys. All of these issues will be addressed after a few more preliminary points are made.




    Encrypting Character, Numbers, and Dates


    Three of the basic data types you may be encrypting and storing are character, number, and dates. As seen in the next example, this can be accomplished fairly easy using the DATA_CRYPTO package. A simple table is created and populated with an encrypted string, number, and date:


    sec_mgr@KNOX10g> CREATE TABLE enc_val_tab   
    2 (charval RAW(32),
    3 dateval RAW(64),
    4 numval RAW(32));

    Table created.

    sec_mgr@KNOX10g> EXEC data_crypto.setkey('EncryptionKey');

    PL/SQL procedure successfully completed.
    sec_mgr@KNOX10g> INSERT INTO enc_val_tab
    2 VALUES (data_crypto.encrypt_char (USER),
    3 data_crypto.encrypt_date (SYSDATE),
    4 data_crypto.encrypt_number (10));

    1 row created.

    sec_mgr@KNOX10g> COMMIT ;

    You can see the data is stored encrypted and can be reconstructed through the decryption process:



    sec_mgr@KNOX10g> COL "Character Data" format a20   
    sec_mgr@KNOX10g> COL "Number Data" format a20
    sec_mgr@KNOX10g> COL "Date Data" format a20
    sec_mgr@KNOX10g> -- Show stored data is encrypted. Have to convert
    sec_mgr@KNOX10g> -- data from RAW to view.
    sec_mgr@KNOX10g> SELECT UTL_RAW.cast_to_varchar2 (charval)
    2 "Character Data",
    3 UTL_RAW.cast_to_varchar2 (numval)
    4 "Number Data",
    5 UTL_RAW.cast_to_varchar2 (dateval)
    6 "Date Data"
    7 FROM enc_val_tab;

    Character Data Number Data Date Data
    -------------------- ----------------- ------------------
    _6-?é?¤+P÷Q¥¯ ÷$ px b?¦$+êUâÿ¦ñ¦ä ¡ùµ¿ò-¦°'>+" ¦-¦

    sec_mgr@KNOX10g> -- Show decrypted values
    sec_mgr@KNOX10g> SELECT data_crypto.decrypt_char (charval)
    2 "Character Data",
    3 TO_CHAR(data_crypto.decrypt_number (numval))
    4 "Number Data",
    5 TO_CHAR
    6 (data_crypto.decrypt_date (dateval),
    7 'DD-MON-RRRR HH24:MI:SS')
    8 "Date Data"
    9 FROM enc_val_tab;

    Character Data Number Data Date Data
    ----------------- ------------- -----------------
    SEC_MGR 10 10-APR-2004 00:00:00


    Everything in the preceding looks functional unless you wanted to preserve the time when you stored the date value.




    Encrypting Dates and the NLS_DATE_FORMAT


    Encrypting dates is a particularly onerous task. The date has to first be converted to a character string, then to a RAW (RAW data is the preferred way to store data). The decryption process must undergo the same steps in reverse order—RAW data is converted to character, then character is converted back to date.


    To do this successfully, you have to ensure your NLS_DATE_FORMAT is set to capture all of the DATE attributes. For example, if the format is DD-MON-RR, then you will lose the time aspect of your date in the data casting operations. The DATA_CRYPTO package will perform the data casting, but you are responsible for ensuring the NLS_DATE_FORMAT is set appropriately.


    This can be a bit more challenging than it might first appear. Notice that the user logged in through SQL*Plus does not get the database’s default NLS_DATE_FORMAT setting:



    sec_mgr@KNOX10g> -- ** Show that current format only shows date not time ** 
    sec_mgr@KNOX10g> COL date_format format a25
    sec_mgr@KNOX10g> SELECT SYS_CONTEXT ('userenv', 'nls_date_format')
    2 DATE_FORMAT,
    3 SYSDATE
    4 FROM DUAL;

    DATE_FORMAT SYSDATE
    ------------- -------------
    DD-MON-RR 10-APR-04

    sec_mgr@KNOX10g> -- The current format was not set by the init.ora parameter
    sec_mgr@KNOX10g> SHOW parameter nls_date_format

    NAME TYPE VALUE
    ------------------- ------------------- ---------
    nls_date_format string DD-MON-RRRR HH24:MI:SS


    You can reset the date format by issuing an ALTER SESSION, but you will have to remember to do this every time prior to encrypting data. An alternative is to create a database logon trigger. Note that the trigger does not have to set everyone’s date format. The following trigger just alters the date format for the SEC_MGR and SCOTT users:



    sec_mgr@KNOX10g> -- Create logon trigger to set date format to capture time. 
    sec_mgr@KNOX10g> -- Consider only setting format for specific users.
    sec_mgr@KNOX10g> CREATE OR REPLACE TRIGGER nls_date_logon
    2 AFTER LOGON ON DATABASE
    3 BEGIN
    4 IF USER IN ('SEC_MGR', 'SCOTT')
    5 THEN
    6 EXECUTE IMMEDIATE
    7 'alter session set nls_date_format=''DD-MON-RRRR HH24:MI:SS'';
    8 END IF;
    9 END;
    10 /

    Trigger created.

    sec_mgr@KNOX10g> -- Show new format
    sec_mgr@KNOX10g> CONN sec_mgr/oracle10g
    Connected.
    sec_mgr@KNOX10g> SELECT SYS_CONTEXT ('userenv', 'nls_date_format')
    2 DATE_FORMAT,
    3 SYSDATE
    4 FROM DUAL;

    DATE_FORMAT SYSDATE
    ----------------- -----------------
    DD-MON-RRRR HH24:MI:SS 10-APR-2004 11:20:07

    sec_mgr@KNOX10g> -- NOTE the format is not set for all users
    sec_mgr@KNOX10g> CONN system/manager
    Connected.
    system@KNOX10g> -- show new format
    system@KNOX10g> SELECT SYS_CONTEXT ('userenv', 'nls_date_format')
    2 DATE_FORMAT,
    3 SYSDATE
    4 FROM DUAL;

    DATE_FORMAT SYSDATE
    -------------- ------------
    DD-MON-RR 10-APR-04


    Once the NLS_DATE_FORMAT has been set appropriately, you’ll be able to preserve the time aspects of your date:



    sec_mgr@KNOX10g> -- Remove previous records. 
    sec_mgr@KNOX10g> TRUNCATE TABLE enc_val_tab;

    Table truncated.

    sec_mgr@KNOX10g> -- Set encryption key
    sec_mgr@KNOX10g> EXEC data_crypto.setkey('EncryptionKey');

    PL/SQL procedure successfully completed.

    sec_mgr@KNOX10g> -- Insert values. NLS_DATE_FORMAT will preserve time.
    sec_mgr@KNOX10g> INSERT INTO enc_val_tab
    2 VALUES (data_crypto.encrypt_char (USER),
    3 data_crypto.encrypt_date (SYSDATE),
    4 data_crypto.encrypt_number (10));

    1 row created.

    sec_mgr@KNOX10g> COMMIT ;

    Commit complete.

    sec_mgr@KNOX10g> -- Show decrypted values
    sec_mgr@KNOX10g> COL "Character Data" format a20
    sec_mgr@KNOX10g> COL "Date Data" format a20
    sec_mgr@KNOX10g> SELECT data_crypto.decrypt_char (charval)
    2 "Character Data",
    3 TO_CHAR(data_crypto.decrypt_number (numval))
    4 "Number Data",
    5 TO_CHAR
    6 (data_crypto.decrypt_date (dateval),
    7 'DD-MON-RRRR HH24:MI:SS')
    8 "Date Data"
    9 FROM enc_val_tab;

    Character Data Number Data Date Data
    ----------------- ------------- ----------
    SEC_MGR 10 10-APR-2004 11:56:47






    Encrypting CLOBs and BLOBs


    The DBMS_CRYPTO package supports the encryption of CLOBs and BLOBs. However, this support is only manifested through the use of procedures. The DATA_CRYPTO package will convert these procedures to functions. For the CLOB data, the ciphertext is also returned as a CLOB.


    In the following example, a table is created that holds CLOBs. This simulates a table that might hold large documents. A CLOB is inserted and then encrypted using a simple SQL update statement.



    sec_mgr@KNOX10g> CREATE TABLE docs  
    2 (doc_id NUMBER(4),
    3 doc CLOB)
    4 /

    Table created.

    sec_mgr@KNOX10g> INSERT INTO docs
    2 VALUES (1, 'This is CLOB data');

    1 row created.

    sec_mgr@KNOX10g> COMMIT ;

    Commit complete.

    sec_mgr@KNOX10g> EXEC data_crypto.setkey('This is the Key');

    PL/SQL procedure successfully completed.

    sec_mgr@KNOX10g> UPDATE docs
    2 SET doc = data_crypto.encrypt (doc);

    1 row updated.

    sec_mgr@KNOX10g> COL Encrypted format a32
    sec_mgr@KNOX10g> COL Decrypted format a32
    sec_mgr@KNOX10g> SELECT doc encrypted, data_crypto.decrypt (doc) decrypted
    2 FROM docs;

    ENCRYPTED DECRYPTED
    ------------------------------ ----------------
    ??N??¦ôa+2IDJ°?-#?_-4o_ÉÖx+| æ{ This is CLOB data


    To test the BLOB encryption, a file is loaded from the file system and stored as a BLOB in the table. The BLOB is then encrypted using a SQL update statement. To verify this process, the BLOB is converted to a CLOB and the value is printed. To begin, a file directory is created and access to the directory is granted to SEC_MGR:


    system@KNOX10g> CREATE OR REPLACE DIRECTORY doc_dir AS  
    2 'C:\tmp\esbd';

    Directory created.

    system@KNOX10g> GRANT ALL ON DIRECTORY doc_dir TO sec_mgr;

    Grant succeeded.

    The DOCS table is modified to hold the BLOB files. The file can be loaded into the table using the DBMS_LOB package. Once loaded, the BLOB can be easily encrypted.



    sec_mgr@KNOX10g> DROP TABLE docs; 

    Table dropped.

    sec_mgr@KNOX10g> CREATE TABLE docs
    2 (doc_id NUMBER(4),
    3 doc BLOB)
    4 /

    Table created.

    sec_mgr@KNOX10g>
    sec_mgr@KNOX10g> DECLARE
    2 l_blob BLOB;
    3 l_bfile BFILE;
    4 BEGIN
    5 INSERT INTO docs
    6 VALUES (1, EMPTY_BLOB ())
    7 RETURNING doc
    8 INTO l_blob;
    9
    10 l_bfile := BFILENAME ('DOC_DIR', 'sample.xml');
    11 DBMS_LOB.fileopen (l_bfile);
    12 DBMS_LOB.loadfromfile (l_blob,
    13 l_bfile,
    14 DBMS_LOB.getlength (l_bfile));
    15 DBMS_LOB.fileclose (l_bfile);
    16 END;
    17 /

    PL/SQL procedure successfully completed.

    sec_mgr@KNOX10g> EXEC data_crypto.setkey('This is the Key');

    PL/SQL procedure successfully completed.

    sec_mgr@KNOX10g> UPDATE docs
    2 SET doc = data_crypto.encrypt (doc);

    1 row updated.

    sec_mgr@KNOX10g> COMMIT ;

    Commit complete.


    Verifying the contents requires a BLOB to CLOB conversion. The BLOB is decrypted, then converted to a CLOB that is printed. The resulting decryption can be compared to the original by using the SQL*Plus Host command:



    sec_mgr@KNOX10g> DECLARE  
    2 l_encrypted_blob BLOB;
    3 l_decrypted_blob BLOB;
    4 l_decrypted_clob CLOB;
    5 l_lang NUMBER := DBMS_LOB.default_lang_ctx;
    6 l_warning NUMBER;
    7 l_t_offset NUMBER := 1;
    8 l_src_offset NUMBER := 1;
    9 BEGIN
    10 SELECT doc
    11 INTO l_encrypted_blob
    12 FROM docs;
    13
    14 DBMS_LOB.createtemporary (l_decrypted_blob, TRUE);
    15 DBMS_LOB.createtemporary (l_decrypted_clob, TRUE);
    16 -- decrypt BLOB
    17 l_decrypted_blob := data_crypto.decrypt (l_encrypted_blob);
    18 -- convert to CLOB for display
    19 DBMS_LOB.converttoclob (l_decrypted_clob,
    20 l_decrypted_blob,
    21 DBMS_LOB.lobmaxsize,
    22 l_t_offset,
    23 l_src_offset,
    24 DBMS_LOB.default_csid,
    25 l_lang,
    26 l_warning
    27 );
    28 DBMS_OUTPUT.put_line (l_decrypted_clob);
    29 END;
    30 /
    <?xml version="1.0"?>
    <document>
    <value>security rawks</value>
    </document>

    PL/SQL procedure successfully completed.


    You can see this data corresponds with the data in your file:


    sec_mgr@KNOX10g> HOST cat C:\tmp\esbd\sample.xml  
    <?xml version="1.0"?>
    <document>
    <value>security rawks</value>
    </document>

    This example used an XML file, which would probably be stored in a CLOB or as an XMLType. You cannot encrypt an XMLType because the database interprets and validates the XML automatically; trying to store encrypted data in this type will fail. BLOBs are ideal for storing true binary data, such as images and audio data. Encrypting this data provides added security.




















    Attack of the Cellular Automata



    [ Team LiB ]









    Attack of the Cellular Automata


    The computer game of Life (first created by Tom Conway) spawned the whole field of study of cellular automata. Will Wright used cellular automata to build SimCity. However, I always thought that it might be fun to fight a war with cellular automata. In this game, the playfield would be a standard array of cellular automata, and the two players would occupy opposite corners. The goal of the game is simple: destroy the enemy's fortress. You do this by launching gliders in his direction. He fends off your gliders with his own. Both of you have the ability to place standard units near your fortress. Gliders are one example; glider factories are another. But you can also place defensive units such as crosses and stars that act like temporary walls. The most important units you can place are "resource factories." These are very expensive, stable units that generate "resource points." These resource points are spent every time you place a unit; if you don't have any resource points, you can't place any units. So you must build and protect your resource factories.


    Using standard game of Life rules, the defense would always have a large advantage over the offense, so some tweaks would be necessary to balance the system. There are plenty of ways to do this. One design solution would be to permit players to modify the rules of individual cells by hitting them with special weapons. For example, one could build a "bomber glider" set to detonate after a specified number of generations; when it explodes, all the cells within the blast radius are modified in such a way as to prevent enemy activity from taking place in those cells for a set period of time. Players could thereby build a safe channel that could be traversed by their own gliders only.


    Bomber gliders could affect their targets in a variety of ways. The basic rules of cellular generation could be altered to render an area dead or overproductive. A dead area blocks activity, while an overproductive region generates lots of dangerous radiation.






      [ Team LiB ]



      2.1 Windows Graphics System Components




      < BACK  NEXT >

      [oR]

      2.1
      WINDOWS GRAPHICS SYSTEM COMPONENTS


      The Windows Application Programming Interface�in other words, the Win32 API�is a huge set of interrelated functions providing different kinds of services to application programs. From the programmer's viewpoint, the Win32 API can be divided into several groups of services:




      • Window Base Services, commonly known as kernel services, which include Microsoft clustering, debugging, error handling, dynamic load library (DLL), process, thread, file, I/O, international features, interprocess communication, performance monitoring, removable storage, security, etc.



      • User Interface Services, commonly known as user services, which include windowing, message queue, dialog box, control, common control, common dialog box, resource, user input, shell, etc.



      • Graphics and Multimedia Services, including broadcast architecture, color management, DirectX, GDI, multimedia, video for windows, still image, OpenGL, Windows Media, etc.



      • COM, OLE, and Active X Services, including Component Object Model (COM), automation, Microsoft Transaction Server, Object Linking and Embedding (OLE), etc.



      • Database and Messaging Services, including Data Access Objects (DOA), the Structured Query Language (SQL) server, Messaging API (MAPI), etc.



      • Networking and Distributed Services, including active directory, message queue, networking, remote procedural call, routing and remote access, Systems Network Architecture (SNA) server, synchronization manager, Telephony API (TAPI), etc.



      • Internet, Intranet, and Extranet Services, including indexing service, Internet Explorer, Microsoft agent, NetShow, scripting, site server, etc.



      • Setup and System Management Services, including configuration, setup, systems management, etc.





      Each group of services is supported by a set of operating-system components. They include Win32 environment subsystem DLLs, user mode drivers, OS system services, and kernel mode drivers. Each group is well worth several good books to uncover the deep knowledge needed to use those services effectively. For example, Jeffrey Richter's Programming Applications for Microsoft Windows, 4th ed. (originally called Advanced Windows), is a masterpiece on windows base services; while Charles Petzold's Programming Windows, 5th ed., focuses on user interface services and GDI services.



      The Graphics and Multimedia Services group of the Win32 API is so huge that it may well take several books to cover adequately. The focus of this book is a very important subset of the Graphics and Multimedia Services, namely GDI and DirectDraw. Let's now take a closer look at the components that support Graphics and Multimedia Services.



      The Win32 graphics APIs are implemented on several platforms: Windows 95/98, WinCE, Windows NT, and the latest Windows 2000. It used to be that NT-based systems provided the best support for GDI, being a true 32-bit implementation, while Windows 95-based systems provided the best support for game programming. But the new Windows 2000 operating system provides the best of both worlds. Windows 2000 makes significant changes to provide hardware acceleration for DirectX/OpenGL, the new still-image API, broadcasting architecture, user mode printer driver, etc. In this book we will focus on the Windows 2000 graphics and multimedia system architecture, while pointing out the differences with Windows 95/98 and Windows NT 3.5/4.0 from time to time.



      Looking top-down at Figure 2-1, as you've seen with the overall operating-system design, the Windows NT/2000 graphics and multimedia system is a layered system. The topmost box contains application programs, which interface with a set of 32-bit user mode system DLLs through the Win32 API. The system DLL layer contains familiar DLLs like GDI32.DLL for the graphics device interface, USER32.DLL for the user interface and window management, KERNEL32.DLL for window base services, etc. Most of the modules in this layer are provided by the operating system, but a few components in this layer rely on help from some user mode drivers provided by hardware vendors. Below this is the system service call gate, which invokes system calls that are served by system service routines in the kernel mode portion of the operating system. The Windows NT/2000 Executive, which lives in kernel address space, provides the graphics engine, I/O manager, video port driver, etc., to support the graphics and multi media system. It needs support from vendor-supplied device drivers, which talk to an array of hardware devices like the bus, video display, and printer with the help of the OS hardware abstraction layer.




      Figure 2-1. Windows 2000 graphics/multimedia system architecture.


      Now let's scan through the user mode portion horizontally. Graphics Device Interface (GDI) and Image Color Management (ICM) provide a device-independent graphics programming interface to application programs. If you're doing printing output, GDI eventually needs to talk to a printer driver, which could be running in user mode in Windows 2000. User mode printer drivers rely heavily on functions provided by the graphics engine, for which they call GDI to pass calls back to the graphics engine. Printing jobs are controlled by the spooler system process, which relies on vendor-customizable components like the print processor, printer monitor, and printer provider.



      DirectX adds a relatively new set of Win32 system DLLs, which implement the DirectX COM interfaces. For actual interfacing with DirectX implementation in kernel address space, DirectX goes through GDI. DirectX is made up DirectDraw, Direct Sound, DirectMusic, DirectInput, DirectPlay, DirectSetup, AutoPlay, and Direct3D. This book covers only the DirectDraw part of DirectX.



      We will cover GDI and the DirectDraw portion of DirectX in much more detail. For the moment, let's just quickly go through the components we will not be covering in this book.



      Multimedia


      The multimedia part of the Win32 API is a continuation from multimedia support started in Windows 3.1. It contains the MCI (media control interface), audio output, multimedia file I/O, joystick, and the multimedia timer. The MCI is defined to control all linear playback media, with functions like load, pause, play, record, stop, resume, etc. There are three types of audio output supported: CD audio, MIDI (musical instrument digital interface), and waveform audio. The Win32 multimedia API is defined in mmsystem.h, with winmm.lib and winmm.dll being its import library, and Win32 system DLL. Winmm.dll relies on installable user mode device drivers for each actual multimedia device. The single major exported function for a multimedia driver, which is itself a 32-bit DLL, is DriverProc. DriverProc handles messages from the multi media system, such as DRV_OPEN, DRV_ENABLE, DRV_CONFIGURE, DRV_CLOSE, etc.



      Note




      To check the multimedia drivers available on your system, open mmdriver.inf under the %SystemRoot%\system32 directory. You can find around a dozen multimedia drivers. For example, mmdrv.dll is for low-level wave, MIDI (musical instrument digital interface), and AUX support (auxillary output device); msacm32.drv is Microsoft Audio Compression Manager; and ir32_32.dll is the Indeo codec, a video compressor/decompressor developed by Intel, which uses a wavelet compression algorithm with MMX support.









      If you wonder how Windows NT/2000 user mode drivers can control hardware like your speaker, they can't do it alone. User mode multimedia drivers rely on a class of kernel mode drivers called kernel streaming drivers, which are capable of controlling hardware directly.



      The Win32 multimedia component is gradually being replaced by corresponding components in DirectX, with better features and higher performance. For example, DirectSound handles waveform-audio sound capture and playback; DirectMusic can capture and play back digital sound samples, including MIDI; DirectInput supports a whole range of input devices including mouse, keyboard, joystick, and other game controllers, as well as force-feedback devices.



      One multimedia function is widely used by non-multimedia Windows applications to get high-resolution timer readings, namely, the timeGetTime() function. It can offer at best 1 ms accuracy, usually better than Get TickCount() (1 ms on Windows 95, 15 ms on Windows NT/2000). For Win32 programs, QueryPerformanceCounter is orders of magnitude more accurate than both timeGetTime and GetTickCount if the CPU supports a high-resolution performance counter. On machines with the Intel Pentium CPU, the high-resolution performance counter is the CPU clock-cycle count we mentioned in Chapter 1. So on a 200-MHz CPU, its accuracy is 5 nanoseconds. But calling Query PerformanceCounter itself is not so high performance; a system call is made to the OS kernel to return the performance counter.




      Video for Windows


      Similar to Win32 multimedia, Win32 Video for Windows has a long history dating back to Windows 3.1. Video for Windows (VFW) provides Win32 API support for video data processing. To be more specific, it supports AVI (audio-video interleaved) file read, write, positioning, and editing, the video compression manager, video capture, and the DrawDib API. Many of VFW's features have been superseded by Direct Show within DirectX.



      The DrawDib API contains functions like DrawDibDraw, DrawDibGetBuffer, DrawDibUpdate, etc. It's similar to the Win32 StretchDIBits call, but provides extra features like calling the right decoder, data streaming, and supposedly higher performance. The first two benefits are from installable multimedia drivers that support different data streams; the third benefit is certainly no match for DirectDraw. Win32 VFW is supported by header file vfw.h, library file vfw32.lib, and runtime DLL msvfw32.dll. The VFW implementation relies on Win32 multimedia implementation.



      Note




      The DrawDib functions are still advertised as providing high-performance image-drawing capabilities which do not rely on GDI and write directly to video memory. While it sounds good, it may no longer be true, especially with Windows NT/2000. On Windows NT/2000, where direct video access is available only through kernel mode driver or DirectX, DrawDibDraw does use the GDI function to draw normal Window DIB, which makes it slower than the GDI DIB drawing function.










      Still Image


      The Still Image (STI) API is a new Microsoft interface for acquiring digital still images from devices like scanners and digital cameras and is available only on Windows 98 and Windows 2000. Clearly STI is a replacement for the older TWAIN standard. (You may wonder why it's not called DirectImage; perhaps it will be soon.) Because it's new, Microsoft has the luxury of using COM interfaces for STI instead of traditional Win32 API function calls. Microsoft STI is made up of a still-image event monitor, vendor-supplied user mode still-image mini-drivers, and a scanner and camera control panel. The still-image event monitor monitors system-wide still image devices and their events. It also keeps track of registered still-image applications, which can be started when an event is detected. A still-image mini-driver detects events from a particular device, notifying the event monitor of activities. It also passes image data from the kernel mode driver to user mode. The scanner and camera control panel allows the user to associate still-image devices with still-image-aware applications. The scanner/camera control panel applet (sticpl.dll), still-image monitor (stimon.dll, stisvc.exe) and imaging applications all use STI COM object (CLSID_Sti), which implements the IStillImage interface, whose instance is created by StiCreateInstance. The STI COM object is implemented by sti.dll, which uses IStiDevice and IStiDeviceControl COM interfaces to control still-image mini-drivers. Windows 98/2000 STI API is supported by header file sti.h, library file sti.lib, the DLLs and EXE mentioned above, and user mode and kernel mode still-image device drivers.




      OpenGL


      The last user mode component in Figure 2-1 is OpenGL. OpenGL is a programming standard for 2D/3D graphics developed by Silicon Graphics, Inc. Its main purpose is to render 2D/3D objects into a frame buffer. OpenGL allows a programmer to describe objects using a group of vertices, each with its coordinate, color, normal, texture coordinate, and edge flag. This allows a description of points, line segments, and 3D surfaces, using OpenGL function calls. The OpenGL graphic control allows specification of transformation, lighting equation coefficients, antialiasing methods, and pixel update operators. The OpenGL render goes through several stages to finally render the data into a frame buffer. The first stage, evaluator, approximates curves and surfaces by evaluating polynomial commands. The second stage, vertex operation and primitive assembly, transforms, lights, and clips vertices. The third stage, rasterization, produces a series of frame-buffer addresses and associated values. The last stage, per-fragment operation, does depth buffering, alpha blending, masking, and other pixel operations in the final frame buffer.



      The Microsoft OpenGL implementation, as seen on Windows NT/2000, adds some extra features to the OpenGL standard. It implements the full set of OpenGL commands, OpenGL Utility (GLU) library, OpenGL Programming Guide Auxiliary Library, Window extension (WGL), per-window pixel format, and double buffering. OpenGL uses three header files in the gl subdirectory of your compiler include directory, including gl.h, glaux.h, and glu.h. The window extension (WGL) part is defined in the GDI header file wingdi.h. OpenGL uses library files opengl.lib and gdi32.lib and runtime DLLs opengl32.dll and gdi32.dll.



      To improve OpenGL performance, its implementation allows for vendor-supplied drivers to perform vendor-specific optimization and direct hardware access. Microsoft provides mini-client architecture (MCD) to facilitate OpenGL drivers. Open GL.dll loads mcd32.dll, the OS-supplied client-side DLL, and an optional vendor-supplied user mode OpenGL driver. You can find your OpenGL driver by searching for OpenGLDrivers in the registry. The MCD client and OpenGL user mode driver uses the GDI function ExtEscape to send commands to the graphics engine and driver in kernel mode. A display driver providing OpenGL optimization is needed to support its MCD portion, with the help of a kernel MCD server in mcdsrv32.dll.



      It's quite common nowadays for graphics card vendors to support DirectDraw, Direct3D, and OpenGL hardware acceleration in a single package. It's always interesting to see different designs for similar purposes, in this case, GDI and OpenGL. Initially, GDI started as a simple graphics programming interface for the hardware the PC industry had during that time, namely, 16- or 256-color EGA or VGA display cards, and monochrome printers. Gradually, GDI added support for device-independent bitmaps, color printers, vector fonts, TrueType fonts, OpenType fonts, 32-bit logical coordinate space, gradient fill, alpha channel, multiple monitor support, multiple terminal support, and so on. GDI is still evolving to GDI+. GDI can really run a small device like a palmtop PC as well as a powerful machine like a workstation. Performance, backward compatibility, and device independence may be the main design goals behind GDI, or the Windows API in general. On the contrary, OpenGL is designed to be a high-end 2D/3D graphics package for creating photorealistic scenes. OpenGL's heavy usage of floating-point calculation needs a high-end machine with lots of memory and horsepower. Various effects like lighting, blending, antialiasing, and fogging would not be effective on a 256-color VGA monitor. Although OpenGL is designed to be device-independent, it's mainly designed to render onto a frame buffer. So OpenGL still has difficulty printing on high-resolution printers. Actually, on Windows NT/2000, GDI is offering an OpenGL printing solution by recording OpenGL commands into a special EMF format and then playing them back to a high-resolution printer device.



      Because of the complexity of building a 2D/3D scene, OpenGL is a higher-level graphics language than GDI. An OpenGL program normally builds a scene in three-dimensional space using vertices, line segments, and polygon surfaces, defines attributes, lighting sources, and viewing angles, and then has the OpenGL engine do all the detailed work. In comparison, GDI is a much more procedural-level language, where the application needs to specify the exact command parameters and sequence to construct a picture. If you want to create a 3D scene, GDI does not help you to calculate 3D depth and remove hidden surfaces. Even Direct3D Immediate Mode is a lower level API when compared with OpenGL.




      Windows Media


      Windows Media is a new addition to the Win32 Graphics/Multimedia system. It contains Windows Media Services, Windows Media Encoder, Windows Media Player Control, and Windows Media Format SDK.



      Windows Media Services provides ActiveX controls and COM APIs to allow Web authors to add streaming media to their web sites and to control, monitor, and managing media broadcasting.



      Windows Media Encoder is primarily responsible for encoding different types of media content into the Windows Media Format stream or file, which can then be delivered through Windows Media Services. Advanced Streaming Format (ASF) is a file container which can contain data in different media formats.



      Windows Media Player Control is an ActiveX control for adding multimedia playback to applications and web pages.



      Windows Media Format SDK is a development kit for reading, writing, and editing Windows Media audio, video, and script files.




      OS Kernel Mode Components


      The user mode graphics and multimedia components have two ways to communicate with the operating system kernel. For GDI, DirectDraw, Direct3D, and OpenGL, user mode calls go through gdi32.dll, which provides the interface to hundreds of system service calls. For interaction with video port drivers and multimedia drivers, user mode calls use the normal file I/O API provided by Windows base services.



      File I/O system services are handled by the I/O manager in the kernel mode Executive, which calls corresponding drivers. GDI, DirectDraw, Direct3D, and OpenGL calls go through the graphics engine, which passes them to individual device drivers.



      The actual operating-system modules involved include NTOSKRNL.EXE (system service dispatching, I/O manager), WIN32K.SYS (graphics engine), MCDSVR32.DLL (MCD server), and HAL.DLL (hardware abstraction layer).



      The Windows NT/2000 kernel Executive, NTOSKRNL.EXE, is the most important piece in the OS kernel. In the graphics system, it's mainly responsible for dispatching graphics-system service calls to the graphics engine, because the graphics engine happens to use the same system service-call mechanism used by other more hard-core system services. The hardware abstraction layer, HAL, provides some service routines to the graphics device driver for things like reading and writing hardware registers. It helps other components of the kernel to be more platform independent. For more discussion on the Executive and HAL, refer to Chapter 1.




      Kernel Mode Drivers


      The Windows NT/2000 graphics and multimedia system relies on quite a number of vendor-supplied drivers to interface with the final hardware.



      The most important driver is the display driver, which should handle GDI, DirectDraw, Direct3D, and MCD for OpenGL support. The display driver is always paired with a video miniport driver, which handles interaction with things like hardware ports. The video miniport driver also needs to support VPE (video port extension to DirectX) and DxApi miniport.



      A font driver is a lesser-known kind of graphics driver, which supplies font glyphs to the graphics engine. For example, ATM (Adobe Type Manager) uses atmfd.dll as a font driver. Font files are loaded into kernel address space by the graphics engine and font drivers.



      The printer driver is similar to the display driver with some more functions to implement. Unlike other drivers, printer drivers normally do not talk directly with printer hardware. Instead, they write the printer-ready data stream back to the spooler in user mode. The spooler passes the data to the print processor and then to the port monitor, which then uses file I/O to call the kernel mode I/O driver. Windows 2000 allows the printer driver to be either user mode DLL or kernel mode DLL.



      Other kernel mode drivers used by the graphics and multimedia system include multimedia drivers like the sound-card driver, and still-image drivers like the scanner driver and camera driver. Windows 2000 DDK provides detailed documentation on kernel streaming drivers (audio, video, video capture) and still-image drivers.



      The quality of the kernel mode device driver is crucial to the stability of the whole operating system. A kernel mode driver has read, write access to the whole kernel mode address space and all privileged CPU instructions. So a buggy kernel mode driver could easily damage important data structures maintained by the operating system, causing the whole system to crash. Therefore, it is critical that any application that includes kernel mode drivers, such as antivirus products, be thoroughly tested to minimize this risk. With Windows 2000, Microsoft ships a Driver Verifier Manager tool (verifier.exe in the system directory) with the OS to help developers verify their drivers.



      This section covered the architecture of the Windows NT/2000 graphics and multi media system, a sophisticated yet structured hierarchy of DLLs, user mode drivers, kernel mode DLLs, and kernel mode drivers. The control flow is much more complicated, as in printing, where control cycles between user mode code and kernel mode code several times. We will leave most of the details to the MSDN library, DDK, and other references, while focusing on a few major components important to common Windows applications. In the remaining sections of this chapter, we will un fold what's in GDI, DirectDraw, the display driver, and the printing system including the printer driver.







      < BACK  NEXT >