Section 16.3. Network Programming in Python

16.3. Network Programming in Python Now that you know all about client/server architecture, sockets, and networking, let us try to bring this concept to Python. The primary module we will be using in this section is the socket module. Found within this module is the socket() function, which is used to create socket objects. Sockets also have their own set of methods, which enable socket-based network communication. 16.3.1. socket() Module Function To create a socket, you must use the socket.socket() function, which has the general syntax: socket(socket_family, socket_type, protocol=0) The socket_family is either AF_UNIX or AF_INET, as explained earlier, and the socket_type is either SOCK_STREAM or SOCK_ DGRAM, also explained earlier. The protocol is usually left out, defaulting to 0. So to create a TCP/IP socket, you call socket.socket() like this: tcpSock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) Likewise, to create a UDP/IP socket you perform: udpSock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) Since there are numerous socket module attributes, this is one of the exceptions where using "from module import *" is somewhat acceptable because of the number of module attributes. If we applied "from socket import *", we bring the socket attributes into our namespace, but our code is shortened considerably, i.e., tcpSock = socket(AF_INET, SOCK_STREAM) Once we have a socket object, all further interaction will occur using that socket object's methods. 16.3.2. Socket Object (Built-in) Methods In Table 16.1, we present a list of the most common socket methods. In the next subsection, we will create both TCP and UDP clients and servers, all of which use these methods. Although we are focusing on Internet sockets, these methods have similar meanings when using Unix sockets. Table 16.1. Common Socket Object Methods Method Description Server Socket Methods s.bind() Bind address (hostname, port number pair) to socket s.listen() Set up and start TCP listener s.accept() Passively accept TCP client connection, waiting until connection arrives (blocking) Client Socket Methods s.connect() Actively initiate TCP server connection s.connect_ex() Extended version of connect() where problems are returned as error codes rather than an exception being thrown General Socket Methods s.recv() Receive TCP message s.send() Transmit TCP message s.sendall() Transmit TCP message completely s.recvfrom() Receive UDP message s.sendto() Transmit UDP message s.getpeername() Remote address connected to socket (TCP) s.getsockname() Address of current socket s.getsockopt() Return value of given socket option s.setsockopt() Set value for given socket option s.close() Close socket Blocking-Oriented Socket Methods s.setblocking() Set blocking or non-blocking mode of socket s.settimeout()[a] Set timeout for blocking socket operations s.gettimeout()[a] Get timeout for blocking socket operations File-Oriented Socket Methods s.fileno() File descriptor of socket s.makefile() Create a file object associated with socket [a] New in Python 2.3. Core Tip: Install clients and servers on different computers to run networked applications In our multitude of examples in this chapter, you will often see code and output referring to host "localhost" or see an IP address of 127.0.0.1. Our examples are running the client(s) and server(s) on the same machine. We encourage the reader to change the hostnames and copy the code to different computers as it is much more fun developing and playing around with code that lets machines talk to one another across the network, and to see network programs that really do work! 16.3.3. Creating a TCP Server We will first present some general pseudocode involved with creating a generic TCP server, then describe in general what is going on. Keep in mind that this is only one way of designing your server. Once you become comfortable with server design, you will be able to modify the pseudocode to operate the way you want it to: ss = socket() # create server socket ss.bind() # bind socket to address ss.listen() # listen for connections inf_loop: # server infinite loop cs = ss.accept() # accept client connection comm_loop: # communication loop cs.recv()/cs.send() # dialog (receive/send) cs.close() # close client socket ss.close() # close server socket # (opt) All sockets are created using the socket.socket() function. Servers need to "sit on a port" and wait for requests, so they all must "bind" to a local address. Because TCP is a connection-oriented communication system, some infrastructure must be set up before a TCP server can begin operation. In particular, TCP servers must "listen" for (incoming) connections. Once this setup process is complete, a server can start its infinite loop. A simple (single-threaded) server will then sit on an accept() call waiting for a connection. By default, accept() is blocking, meaning that execution is suspended until a connection arrives. Sockets do support a non-blocking mode; refer to the documentation or operating systems textbooks for more details on why and how you would use non-blocking sockets. Once a connection is accepted, a separate client socket is returned [by accept()] for the upcoming message interchange. Using the new client socket is similar to handing off a customer call to a service representative. When a client eventually does come in, the main switchboard operator takes the incoming call and patches it through, using another line to the right person to handle their needs. This frees up the main line, i.e., the original server socket, so that the operator can resume waiting for new calls (client requests) while the customer and the service representative he or she was connected to carry on their own conversation. Likewise, when an incoming request arrives, a new communication port is created to converse directly with that client while the main one is free to accept new client connections. Core Tip: Spawning threads to handle client requests We do not implement this in our examples, but it is also fairly common to hand a client request off to a new thread or process to complete the client processing. The SocketServer module, a high-level socket communication module written on top of socket, supports both threaded and spawned process handling of client requests. We refer the reader to the documentation to obtain more information about the SocketServer module as well as the exercises in Chapter 17, Multithreaded Programming. Once the temporary socket is created, communication can commence, and both client and server proceed to engage in a dialog of sending and receiving using this new socket until the connection is terminated. This usually happens when one of the parties either closes its connection or sends an empty string to its partner. In our code, after a client connection is closed, the server goes back to wait for another client connection. The final line of code, where we close the server socket, is optional. It is never encountered since the server is supposed to run in an infinite loop. We leave this code in our example as a reminder to the reader that calling the close() method is recommended when implementing an intelligent exit scheme for the server, for example, a handler that detects some external condition whereby the server should be shut down. In those cases, a close() method call is warranted. In Example 16.1, we present tsTserv.py, a TCP server program that takes the data string sent from a client and returns it timestamped (format: "[timestamp]data") back to the client. ("tsTserv" stands for timestamp TCP server. The other files are named in a similar manner.) Example 16.1. TCP Timestamp Server (tsTserv.py) Creates a TCP server that accepts messages from clients and returns them with a timestamp prefix. 1 #!/usr/bin/env python 2 3 from socket import * 4 from time import ctime 5 6 HOST = '' 7 PORT = 21567 8 BUFSIZ = 1024 9 ADDR = (HOST, PORT) 10 11 tcpSerSock = socket(AF_INET, SOCK_STREAM) 12 tcpSerSock.bind(ADDR) 13 tcpSerSock.listen(5) 14 15 while True: 16 print 'waiting for connection...' 17 tcpCliSock, addr = tcpSerSock.accept() 18 print '...connected from:', addr 19 20 while True: 21 data = tcpCliSock.recv(BUFSIZ) 22 if not data: 23 break 24 tcpCliSock.send('[%s] %s' % ( 25 ctime(), data)) 26 27 tcpCliSock.close() 28 tcpSerSock.close() Line-by-Line Explanation Lines 14 After the Unix start-up line, we import time.ctime() and all the attributes from the socket module. Lines 613 The HOST variable is blank, an indication to the bind() method that it can use any address that is available. We also choose an arbitrarily random port number, which does not appear to be used or reserved by the system. For our application, we set the buffer size to 1K. You may vary this size based on your networking capability and application needs. The argument for the listen() method is simply a maximum number of incoming connection requests to accept before connections are turned away or refused. The TCP server socket (tcpSerSock) is allocated on line 11, followed by the calls to bind the socket to the server's address and to start the TCP listener. Lines 1528 Once we are inside the server's infinite loop, we (passively) wait for a connection. When one comes in, we enter the dialog loop where we wait for the client to send its message. If the message is blank, that means that the client has quit, so we would break from the dialog loop, close the client connection, and go back to wait for another client. If we did get a message from the client, then we format and return the same data but prepended with the current timestamp. The final line is never executed, but is there as a reminder to the reader that a close() call should be made if a handler is written to allow for a more graceful exit, as we discussed before. 16.3.4. Creating a TCP Client Creating a client is much simpler than a server. Similar to our description of the TCP server, we will present the pseudocode with explanations first, then show you the real thing. cs = socket() # create client socket cs.connect() # attempt server connection comm_loop: # communication loop cs.send()/cs.recv() # dialog (send/receive) cs.close() # close client socket As we noted before, all sockets are created using socket.socket(). Once a client has a socket, however, it can immediately make a connection to a server by using the socket's connect() method. When the connection has been established, then it can participate in a dialog with the server. Once the client has completed its transaction, it may close its socket, terminating the connection. We present the code for tsTclnt.py in Example 16.2; it connects to the server and prompts the user for line after line of data. The server returns this data timestamped, which is presented to the user by the client code. Example 16.2. TCP Timestamp Client (tsTclnt.py) Creates a TCP client that prompts the user for messages to send to the server, gets them back with a timestamp prefix, and displays the results to the user. 1 #!/usr/bin/env python 2 3 from socket import * 4 5 HOST = 'localhost' 6 PORT = 21567 7 BUFSIZ = 1024 8 ADDR = (HOST, PORT) 9 10 tcpCliSock = socket(AF_INET, SOCK_STREAM) 11 tcpCliSock.connect(ADDR) 12 13 while True: 14 data = raw_input('> ') 15 if not data: 16 break 17 tcpCliSock.send(data) 18 data = tcpCliSock.recv(BUFSIZ) 19 if not data: 20 break 21 print data 22 23 tcpCliSock.close() Line-by-Line Explanation Lines 13 After the Unix startup line, we import all the attributes from the socket module. Lines 511 The HOST and PORT variables refer to the server's hostname and port number. Since we are running our test (in this case) on the same machine, HOST contains the local hostname (change it accordingly if you are running your server on a different host). The port number PORT should be exactly the same as what you set for your server (otherwise there won't be much communication[!]). We also choose the same buffer size, 1K. The TCP client socket (tcpCliSock) is allocated on line 10, followed by (an active) call to connect to the server. Lines 1323 The client also has an infinite loop, but it is not meant to run forever like the server's loop. The client loop will exit on either of two conditions: the user enters no input (lines 14-16), or the server somehow quit and our call to the recv() method fails (lines 18-20). Otherwise, in a normal situation, the user enters in some string data, which is sent to the server for processing. The newly timestamped input string is then received and displayed to the screen. 16.3.5. Executing Our TCP Server and Client(s) Now let us run the server and client programs to see how they work. Should we run the server first or the client first? Naturally, if we ran the client first, no connection would be possible because there is no server waiting to accept the request. The server is considered a passive partner because it has to establish itself first and passively wait for a connection. A client, on the other hand, is an active partner because it actively initiates a connection. In other words: Start the Server First (Before Any Clients Try to Connect). In our example running of the client and server, we use the same machine, but there is nothing to stop us from using another host for the server. If this is the case, then just change the hostname. (It is rather exciting when you get your first networked application running the server and client from different machines!) We now present the corresponding (input and) output from the client program, which exits with a simple RETURN (or Enter key) keystroke with no data entered: $ tsTclnt.py > hi [Sat Jun 17 17:27:21 2006] hi > spanish inquisition [Sat Jun 17 17:27:37 2006] spanish inquisition > $ The server's output is mainly diagnostic: $ tsTserv.py waiting for connection... ...connected from: ('127.0.0.1', 1040) waiting for connection... The "... connected from ..." message was received when our client made its connection. The server went back to wait for new clients while we continued receiving "service." When we exited from the server, we had to break out of it, resulting in an exception. The best way to avoid such an error is to create a more graceful exit, as we have been discussing. Core Tip: Exit gracefully and call server close() method One way to create this "friendly" exit is to put the server's while loop inside the except clause of a TRy-except statement and monitor for EOFError or KeyboardInterrupt exceptions. Then in the except clause, you can make a call to close the server's socket. The interesting thing about this simple networked application is that we are not only showing how our data take a round trip from the client to the server and back to the client, but we also use the server as a sort of "time server," because the timestamp we receive is purely from the server. 16.3.6. Creating a UDP Server UDP servers do not require as much setup as TCP servers because they are not connection-oriented. There is virtually no work that needs to be done other than just waiting for incoming connections. ss = socket() # create server socket ss.bind() # bind server socket inf_loop: # server infinite loop cs = ss.recvfrom()/ss.sendto()# dialog (receive/send) ss.close() # close server socket As you can see from the pseudocode, there is nothing extra other than the usual create-the-socket and bind it to the local address (host/port pair). The infinite loop consists of receiving a message from a client, returning a timestamped one, then going back to wait for another message. Again, the close() call is optional and will not be reached due to the infinite loop, but it serves as a reminder that it should be part of the graceful or intelligent exit scheme we've been mentioning. One other significant different between UDP and TCP servers is that because datagram sockets are connectionless, there is no "handing off" of a client connection to a separate socket for succeeding communication. These servers just accept messages and perhaps reply. You will find the code to tsUserv.py in Example 16.3, a UDP version of the TCP server seen earlier. It accepts a client message and returns it to the client timestamped. Example 16.3. UDP Timestamp Server (tsUserv.py) Creates a UDP server that accepts messages from clients and returns them with a timestamp prefix. 1 #!/usr/bin/env python 2 3 from socket import * 4 from time import ctime 5 6 HOST = '' 7 PORT = 21567 8 BUFSIZ = 1024 9 ADDR = (HOST, PORT) 10 11 udpSerSock = socket(AF_INET, SOCK_DGRAM) 12 udpSerSock.bind(ADDR) 13 14 while True: 15 print 'waiting for message...' 16 data, addr = udpSerSock.recvfrom(BUFSIZ) 17 udpSerSock.sendto('[%s] %s' % ( 18 ctime(), data), addr) 19 print '...received from and returned to:', addr 20 21 udpSerSock.close() Line-by-Line Explanation Lines 14 After the Unix startup line, we import time.ctime() and all the attributes from the socket module, just like the TCP server setup. Lines 612 The HOST and PORT variables are the same as before, and for all the same reasons. The call socket() differs only in that we are now requesting a datagram/UDP socket type, but bind() is invoked in the same way as in the TCP server version. Again, because UDP is connectionless, no call to "listen() for incoming connections" is made here. Lines 1421 Once we are inside the server's infinite loop, we (passively) wait for a message (a datagram). When one comes in, we process it (by adding a timestamp to it), then send it right back and go back to wait for another message. The socket close() method is there for show only, as indicated before. 16.3.7. Creating a UDP Client Of the four highlighted here in this section, the UDP client is the shortest bit of code that we will look at. The pseudocode looks like this: cs = socket() # create client socket comm_loop: # communication loop cs.sendto()/cs.recvfrom() # dialog (send/receive) cs.close() # close client socket Once a socket object is created, we enter the dialog loop of exchanging messages with the server. When communication is complete, the socket is closed. The real client code, tsUclnt.py, is presented in Example 16.4. Example 16.4. UDP Timestamp Client (tsUclnt.py) Creates a UDP client that prompts the user for messages to send to the server, gets them back with a timestamp prefix, and displays them back to the user. 1 #!/usr/bin/env python 2 3 from socket import * 4 5 HOST = 'localhost' 6 PORT = 21567 7 BUFSIZ = 1024 8 ADDR = (HOST, PORT) 9 10 udpCliSock = socket(AF_INET, SOCK_DGRAM) 11 12 while True: 13 data = raw_input('> ') 14 if not data: 15 break 16 udpCliSock.sendto(data, ADDR) 17 data, ADDR = udpCliSock.recvfrom(BUFSIZ) 18 if not data: 19 break 20 print dataudpCliSock.close() 21 22 udpCliSock.close() Line-by-Line Explanation Lines 13 After the Unix startup line, we import all the attributes from the socket module, again, just like in the TCP version of the client. Lines 510 Because we are running the server on our local machine again, we use "localhost" and the same port number on the client side, not to mention the same 1K buffer. We allocate our socket object in the same way as the UDP server. Lines 1222 Our UDP client loop works in almost the exact manner as the TCP client. The only difference is that we do not have to establish a connection to the UDP server first; we simply send a message to it and await the reply. After the timestamped string is returned, we display it to the screen and go back for more. When the input is complete, we break out of the loop and close the socket. 16.3.8. Executing Our UDP Server and Client(s) The UDP client behaves the same as the TCP client: $ tsUclnt.py > hi [Sat Jun 17 19:55:36 2006] hi > spam! spam! spam! [Sat Jun 17 19:55:40 2006] spam! spam! spam! > $ Likewise for the server: $ tsUserv.py waiting for message... ...received from and returned to: ('127.0.0.1', 1025) waiting for message... In fact, we output the client's information because we can be receiving messages from multiple clients and sending replies, and such output helps by telling us where messages came from. With the TCP server, we know where messages come from because each client makes a connection. Note how the messages says, "waiting for message" as opposed to "waiting for connection." 16.3.9. socket Module Attributes In addition to the socket.socket() function which we are now familiar with, the socket module features many more attributes that are used in network application development. Some of the most popular ones are shown in Table 16.2. For more information, we refer you to the socket Module documentation in the Python Library Reference. Table 16.2. socket Module Attributes Attribute Name Description Data Attributes AF_UNIX, AF_INET, AF_INET6[a] Socket address families supported by Python SO_STREAM, SO_DGRAM Socket types (TCP = stream, UDP = datagram) has_ipv6[b] Boolean flag indicating whether IPv6 is supported Exceptions error Socket-related error herror[a] Host and address-related error gaierror[a] Address-related error timeout[b] Timeout expiration Functions socket() Create a socket object from the given address family, socket type, and protocol type (optional) socketpair()[c] Create a pair of socket objects from the given address family, socket type, and protocol type (optional) fromfd() Create a socket object from an open file descriptor Data Attributes   ssl()[d] Initiates a Secure Socket Layer connection over socket; does not perform certificate validation getaddrinfo()[a] Gets address information getfqdn()[e] Returns fully qualified domain name gethostname() Returns current hostname gethostbyname() Maps a hostname to its IP address gethostbyname_ex() Extended version of gethostbyname() returning hostname, set of alias hostnames, and list of IP addresses gethostbyaddr() Maps an IP address to DNS info; returns same 3-tuple as gethostbyname_ex() getprotobyname() Maps a protocol name (e.g. 'tcp') to a number getservbyname()/getservbyport() Maps a service name to a port number or vice-versa; a protocol name is optional for either function ntohl()/ntohs() Converts integers from network to host byte order htonl()/htons() Converts integers from host to network byte order inet_aton()/inet_ntoa() Convert IP address octet string to 32-bit packed format or vice versa (for IPv4 addresses only) inet_pton()/inet_ntop()[b] Convert IP address string to packed binary format or vice versa (for both IPv4 and IPv6 addresses) geTDefaulttimeout()/setdefaulttimeout()[b] Return default socket timeout in seconds (float); set default socket timeout in seconds (float) [a] New in Python 2.2. [b] New in Python 2.3. [c] New in Python 2.4. [d] New in Python 1.6. [e] New in Python 2.0.