|2020ok Directory of FREE Online Books and FREE eBooks|
Text Processing In Python
by David Mertz
(Respecting the intellectual property of others is utmost important to us, we make every effort to make sure we only link to legitimate sites, such as those sites owned by authors and publishers. If you have any questions about these links, please contact us.)
Text Processing in Python
describes techniques for manipulation of text using the
Python programming language. At the broadest level, text processing is simply
taking textual information and doing something with it. This might be
restructuring or reformatting it, extracting smaller bits of information from it,
or performing calculations that depend on the text. Text processing is arguably
what most programmers spend most of their time doing. Because Python is
clear, expressive, and object-oriented it is a perfect language for doing text
processing, even better than Perl. As the amount of data everywhere continues
to increase, this is more and more of a challenge for programmers. This book is
not a tutorial on Python. It has two other goals: helping the programmer get
the job done pragmatically and efficiently; and giving the reader an
understanding - both theoretically and conceptually - of why what works works
and what doesn't work doesn't work. Mertz provides practical pointers and tips
that emphasize efficent, flexible, and maintainable approaches to the textprocessing
tasks that working programmers face daily.
From the Back Cover
Text Processing in Python is an example-driven, hands-on tutorial that carefully teaches programmers how to accomplish numerous text processing tasks using the Python language. Filled with concrete examples, this book provides efficient and effective solutions to specific text processing problems and practical strategies for dealing with all types of text processing challenges.
Text Processing in Python begins with an introduction to text processing and contains a quick Python tutorial to get you up to speed. It then delves into essential text processing subject areas, including string operations, regular expressions, parsers and state machines, and Internet tools and techniques. Appendixes cover such important topics as data compression and Unicode. A comprehensive index and plentiful cross-referencing offer easy access to available information. In addition, exercises throughout the book provide readers with further opportunity to hone their skills either on their own or in the classroom. A companion Web site (http://gnosis.cx/TPiP) contains source code and examples from the book.
Here is some of what you will find in thie book:
About the Author
David Mertz came to writing about programming via the unlikely route of first being a humanities professor. Along the way, he was a senior software developer, and now runs his own development company, Gnosis Software ("We know stuff!"). David writes regular columns and articles for IBM developerWorks, Intel Developer Network, O'Reilly ONLamp, and other publications.
Excerpt. © Reprinted by permission. All rights reserved.
Beautiful is better than ugly.
0.1 What is Text Processing?
At the broadest level text processing is simply taking textual information and doing something with it. This doing might be restructuring or reformatting it, extracting smaller bits of information from it, algorithmically modifying the content of the information, or performing calculations that depend on the textual information. The lines between "text" and the even more general term "data" are extremely fuzzy; at an approximation, "text" is just data that lives in forms that people can themselves read—at least in principle, and maybe with a bit of effort. Most typically computer "text" is composed of sequences of bits which have a "natural" representation as letters, numerals and symbols; and most often such text is delimited (if delimited at all) by symbols and formatting that can be easily pronounced as "next datum."
The lines are fuzzy, but the data that seems least like text—and that, therefore this particular book is least concerned with—is the data that makes up "multimedia" (pictures, sounds, video, animation, etc.) and data that makes up UI "events" (draw a window, move the mouse, open an application, etc.). Like I said, the lines are fuzzy, and some representations of the most non-textual data are themselves pretty textual. But in general, the subject of this book is all the stuff on the near side of that fuzzy line.
Text processing is arguably what most programmers spend most of their time doing. The information that lives in business software systems mostly comes down to collections of words about the application domain—maybe with a few special symbols mixed in. Internet communications protocols consist mostly of a few special words used as headers, a little bit of constrained formatting, and message bodies consisting of additional wordish texts. Configuration files, log files, CSV and fixed-length data files, error files, documentation, and source code itself, are all just sequences of words with bits of constraint and formatting applied.
Programmers and developers spend so much time with text processing, that it is easy to forget that that is what we are doing. The most common text processing application is probably your favorite text editor. Beyond simple entry of new characters, text editors perform such text processing tasks as search/replace and copy/paste, which—given guided interaction with the user—accomplishes sophisticated manipulation of textual sources. Many text editors go farther than these simple capabilities, and include their own complete programming systems (usually called "macro processing"); in those cases where editors include "Turing-complete" macro languages, text editors suffice, in principle, to accomplish anything that the examples in this book can.
After text editors, a variety of text processing tools are widely used by developers. Tools like "File Find" under Windows, or "grep" on Unix (and other platforms) perform the basic chore of locating text patterns. "Little languages" like sed and awk perform basic text manipulation (or even non-basic). A large number of utilities—especially in Unix-like environments—perform small custom text processing tasks:
At the top of the text processing food chain are general purpose programming languages, such as Python. I wrote this book on Python in large part because Python is such a clear, expressive, and general purpose language. But for all Python's virtues, text editors and "little" utilities will always have an important place for developers "getting the job done." As simple as Python is, it is still more complicated than you need to achieve many basic tasks. But once you get past the very simple, Python is a perfect language for making the difficult things possible (and it is also good at making the easy things simple).
0.2 The Philosophy of Text Processing
Hang around any Python discussion groups for a little while, and you will certainly be dazzled by the contributions of the Python developer, Tim Peters (and by a number of other Pythonistas). His "Zen of Python" captures much of the reason that I choose Python as the language in which to solve most programming tasks that are presented to me. But to understand what is most special about text processing as a programming task, it is worth turning to Perl creator Larry Wall's cardinal virtues of programming: Laziness, impatience, hubris.
What sets text processing most clearly apart from other tasks computer programmers accomplish is the frequency with which we perform text processing on an ad hoc or "one-shot" basis. One rarely bothers to create a one-shot GUI interface for a program. You even less frequently performs a one-shot normalization of a relational database. But every programmer with a little experience has had numerous occasions where she has received a trickle of textual information (or maybe a deluge of it) from another department, from a client, from a developer working on a different project, or from data dumped out of a DBMS; the problem in such cases is always to "process" the text so that it is usable for our own project, program, database, or work unit. Text processing to the rescue. This is where the virtue of impatience first appears—we just want the stuff processed, right now!
But text-processing tasks that were obviously one-shot tasks that we knew we would never need again have a habit of coming back like restless ghosts. It turns out that that client needs to update the one-time data they sent last month. Or the boss decides that she would really like a feature of that text summarized in a slightly different way. The virtue of laziness is our friend here—with our foresight not to actually delete those one-shot scripts, we have them available for easy reuse and/or modification when the need arises.
Enough is not enough, however. That script you reluctantly used a second time turns out to be quite similar to a more general task you will need to perform frequently, perhaps even automatically. You imagine that with only a slight amount of extra work you can generalize and expand the script, maybe add a little error checking and some runtime options while you are at it; and do it all in time and under budget (or even as a side project, off the budget). Obviously, this is the voice of that greatest of programmers' virtues: hubris.
The goal of this book is to make its readers a little lazier, a smidgeon more impatient, and a whole bunch more hubristic. Python just happens to be the language best suited to the study of virtue.
0.3 What You'll Need to Use This Book
This book is ideally suited for programmers who are a little bit familiar with Python, and whose daily tasks involve a fair amount of text processing chores. Programmers who have some background in other programming languages—especially with other "scripting" languages—should be able to pick up enough Python to get going by reading Appendix A.
While Python is a rather simple language at heart, this book is not intended as a tutorial on Python for non-programmers. Instead, this book is about two other things: getting the job done, pragmatically and efficiently; and understanding why what works works and what doesn't work doesn't work, theoretically and conceptually. As such, we hope this book can be useful both to working programmers and to students of programming at a level just past the introductory.
Many sections of this book are accompanied by problems and exercises, and these in turn often pose questions for users. In most cases, the answers to the listed questions are somewhat open-ended—there are no simple right answers. I believe that working through the provided questions will help both self-directed and instructor-guided learners; the questions can typically be answered at several levels, and often have an underlying subtlety. Instructors who wish to use this text are encouraged to contact the author for assistance in structuring a curriculum involving it. All readers are encouraged to consult the book's web site to see possible answers provided by both the author and other readers; additional related questions will be added to the web site over time, along with other resources.
The Python language itself is conservative. Almost every Python script written ten years ago for Python 1.0 will run fine in Python 2.3+. However, as versions improve, a certain number of new features have been added. The most significant changes have matched the version number changes—Python 2.0 introduced list comprehension's, augmented assignments, Unicode support, and a standard XML package. Many scripts written in the most natural and efficient manner using Python 2.0+ will not run without changes in earlier versions of Python.
The general target of this book will be users of Python 2.1+, but some 2.2+ specific features will be utilized in examples. Maybe half the examples in this book will run fine on Python 1.5.1+ (and slightly fewer wit...
Related Free eBooks