How Oracle Works

Why Oracle Works the Way it Does Introduction
Dratz Apr 25, 2006 | Comments (6) inShare
0 I have been asked by friends and coworkers a lot recently about the Oracle database. I have been involved in some major recovery and tuning efforts lately and some of them were a bit surprised because my job is no longer primarily as a DBA. I explained to them what I have explained to so many people: if you understand the basics, you can figure out everything else. So I decided to start this series and this first part is just going to explain my goals and motivations. Subsequent posts will delve into the architectural guts of the Oracle RDBMS. I have worked with a lot of technology and a lot of software in my life (even predating my career) and I can say without hestitaion that the Oracle RDBMS is the most sophisticated software application in the history of the world. No, I'm not saying everything else sucks or that Oracle is even the right database for you to use (I hate when people ask me what the best database is without expressing their needs or ciscumstances). I'm just saying one thing about Oracle that I cannot say about much other software: It is exceedingly sophisticated, elegant, and powerful. Lots of software may be, but not to the extent that I think the Oracle RDBMS is. Perhaps because I've never had a day of Oracle training (but did get my OCP) and had to struggle through learning it on my own (literally) with only some books (no askTom or Google back then), I've come to appreciate how easy it is to correctly administer a good Oracle database. Easy? Yes, without question. And I hope to show you how. I am not going to compare Oracle to the other competent databases out there, I'm not going to tell you why you should buy Oracle, I'm not going to tell you that you should convert all of your databases to Oracle. There's something else I hope not to do. I'm not going to tell you how to do anything. I'm not going to provide step-by-step instructions to install, recover, or tune an Oracle
database. I'm not going to suggest best practices. Too many people get into trouble following instructions that were designed to address a specific problem and may not work as well for another specific problem. What I hope to do is explain why Oracle works the way it does so that when you find step-by-step instructions, YOU can determine if you think those instructions will work or not. I don't know how many meetings I've walked into where people sat around a table debating how Oracle worked as if once they agreed Oracle would simply comply and work whichever way the vote went. Oracle is not free of bugs. Like any software, they are constantly releasing patches to address different issues. But none of these patches really affect the underlying architecture or intent of how Oracle works. I am going to try and stay with core concepts and realities and not focus on the different versions of Oracle except when the changes are dramatic in terms of what I'm addressing. I will simply try to make the concepts honest as of Oracle 8.1.7 and up. Most of this will apply to earlier versions as well, but if you're software is that old, just upgrade already. I am not offering free consulting experience or any how-to questions. When I have those kinds of questions, I ask Lewis Cunningham, Thomas Kyte, and Steve Adams among others like Mr. Google. I don't, however, waste their time on questions I can look up in a book. Because I understand the fundamentals of how Oracle works, I like to think that I can ask intelligent questions and respect their time. So, if nothing else, hopefully this series will help you make better (and less frequent) use of those gentlemen. I will get you started by asking you remember the simplicity of Oracle's nested layering of physical and logical constructs. I am going to start this post at the bottom and work my way up. Though I will occasionally pop to higher constructs (like rows and columns), I want to start at the bottom and keep referring new ideas back to how they relate to these primative layers. This series is really designed for junior Oracle DBA's, wanna-be DBA's, DBA's of other RDBMS platforms, developers who want to find ways to build better database applications, and architects who want to design better databases. I will not be offended if managers want to participate so they have a better understanding of what their staff does for a living. Ok, so the next post will start with the DATAFILES on the physical side and the TABLESPACES that use them on the logical side.
----------------------------Some senior DBA's may have questions or concerns about what I'm presenting and how. There will be gaps between my posts because I have to not only check them for accuracy but also for clarity (no point in explaining the right thing the wrong way). I may explain some things a little differently than you've seen before, but they will be cleared up in the course of posts. This is just the way I've been training new DBA's for years.
Why Oracle Works the Way it Does (1) Datafiles and Tablespaces
2 Now, the smallest unit Oracle really messes with is the "data block". I'm going to skip over them for now because I think they'll make more sense after we discuss datafiles and tablespaces. You'll remember (because I asked you to) that Oracle employs an elegant weaving of physical and logical layers. As we go through them, some may wonder "why do all these layers exist?" All these layers (I hope to discuss the most important ones) exist for: performance, scalability, concurrency, and managability. Ok, you have a wonderful database with tables and indexes and then, what do those lousy users do? They start using your database and start pumping more and more data into it. Where does all that data go? That's what we are going to talk about today. -------------------------------When I use MS Word to write a document, I have 2 options when I'm done working on it: I can throw it all away or I can save it to disk. If I choose to save it to disk, I don't just save it randomly anywhere on the disk, I save it someplace where I can easily find it again. I save it to a directory or folder. I may have multiple directories, one for technical documents, one for various versions of my resume, one for the exciting screenplays I write that will never be made into movies. I can have as many documents as I want in those directories*, I'm only limited by the capacity of the disk. That makes sense, right? Ok, so every document I save has a definite size to it. I can browse the directory and see how many files I have, how big each of them are, and what the total size of all my files in a directory is. So how big is my directory? Outside of some minimal space to hold the specification, it's really just a logical construct. The directory itself doesn't really have any size of its own right and is defined by the size of all of the files in it? How big can it get? As big as it gets with files before I run out of disk space. Congratulations, you now understand the basic concepts behind DATAFILES and TABLESPACES in Oracle. They don't work exactly that way, but pretty close. Word doc= datafile directory = tablespace
( A tablespace is in reality not exactly like a OS level directory (it just functions the same kind of way). Datafiles can only belong to a single tablespace, but can be physically located in more than one OS directory.) All objects in an Oracle database are assigned to a tablespace, and the tablespace knows which datafiles it has to actually store the objects and data. You don't specify the size of a tablespace, it doesn't really have a max size (unless you want to do the math in 10g to finally come up with thousands of petabytes). You do, however, specify the size of datafiles. This is like telling a Word doc that you plan on saving a million words in your document so go ahead and reserve that space now. This preallocation is nice, because you don't suffer the overhead of trying to find new space to add every time you type in a new word. So, what do you do when you start approaching your million word limit and see that you are going to need space for a lot more words? Well, you have two basic options: you can add another file and let the additional words go there, or you can let the files you have just grow to make room. In Oracle, if you want you your datafiles to grow as needed, you use the AUTOEXTEND keyword when creating or altering the datafile (just make sure to set the max size so you don't chew up all the disk space). So which is better, several smaller files or fewer larger files? Generically, neither is better but it is often easier to herd a couple of cats than a hundred. And if you're going to be a good DBA, you want to make your life as easy as you can. If you oversize your datafiles, you could be wasting space (and time). If you undersize them and don't tell them how to grow (using AUTOEXTEND), you will be getting paged a lot with warnings that your tablespace is almost out of room. Everything you do as a DBA should be done for a reason. If you choose to do no analysis of your environment and your needs, you'll be working a lot harder than you need to. How many tablespaces should you have? Well, there are a couple of tablespaces every Oracle database requires to function, but we'll discuss those in the future. All we care about now is you data. You probably already know about tables and indexes, right? Tables hold data and indexes (can) make it a lot quicker and easier to access that data. A lot (and I mean a lot) of people like to separate their tables from their indexes and put them in different tablespaces (with datafiles on different disks). There's nothing inherently wrong with that if you have a reason to do it, but I want to mention one fallacy that too many people take as not only valid but as an absloute truth: that you MUST separate your you indexes from your tables. The theory goes that you want to reduce I/O contention on your disks by not fighting
for read control between scanning indexes and scanning tables(rows). This is wrong on at least three fronts: -your indexes will often (not always) be cached in memory anyway -your disk heads are constantly bouncing around unless maybe you have a single user -it means you don't understand how sql queries work I don't want to get into it too much here, but it is germain to the discussion of tablespaces. When you issue a SELECT statement, the query optimizer will analyze your statement and decide how it wants to get the data. It will decide to use an index or not to use one. If it does use an index, it has to read the index BEFORE it knows where to find the row(s) in the table. First one THEN the other. Access for this query will not simultaneously read the index and the table. It can't. So split them into different tablespaces if you want to, but don't let anyone tell you that you HAVE to. Some people have lot's of tablespaces, some people only have one (outside of the required ones). Well talk about some possible reasons for each approach in the future. As long as you understand what tablespaces and datafiles are, we can move on for now. The main thing to be able to do at the end of this series is to be able to make intelligent choices. And a lot of times that just means trying something and seeing how it works. Never be afraid to change when you need to. How does all of this really work? All of you applications data and other objects will reside inside of a tablespace and it's related datafiles. If Oracle were not super friendly and easy to use, it would make you manage how that data is stored manually. Fortunately it is super friendly and easy to use. It's even flexible. In older versions of Oracle you had to use Dictionary Managed Tablespaces (DMT's) (we'll talk about the data dictionary later). This was fine and still is, but it's not the best you can do. Since Oracle 8i you can take advantage of Locally Managed Tablespaces(LMT's). LMT's offer so many advantages to DMT's (mainly because of things like allocation overhead and fragmentation) that few people choose or can find good reasons to use DMT's nowadays. It doesn't mean you're an idiot if you use DMT's, it just means people will think you are. We have a lot more to discuss with datafiles and tablespaces, but this is enough to get us started and ready for the next topic. ----------------------* Oracle 10g has introduced a new tablespace/datafile concept called bigfile, but I'm going to save that discussion unti we get to OLTP vs. DSS systems.
Segments and Extents

0 Ok, now that we have a basic understanding of datafiles and tablespaces, we are one step closer to discussing data blocks and something I haven't mentioned called "headers". But before we get there, we need to bridge the gap between D&T and blocks. That bridge comes in the form of two logical constructs that build off of the block physical layer. Those constructs are SEGMENTS and EXTENTS. An easy way to think of these two are as mapping tools. They don't actually store data, they just keep it organized and accessible. In that way, they are not very dissimilar from tablespaces (just at a more granular level). SEGMENTS When you type in the syntax to create a table, one thing you have to provide (among others) is a table name. That table name along with the "create table" keywords gives Oracle everything it needs to create a segment. A segment is registered with the tablespace as the traffic cop for the table. Like a tablespace, a segment has no real space of its own; it is the sum of it's parts. However, one thing you can do with a segment is put a limit of how big it can ever get by limiting its extents. The segment is a critical part of the RDBMS, but is not one you have lose a lot of sleep over right now. You will, over time, find that you want to give some better instructions or definition to your segments, but most of these can be accomplished via ALTER TABLE statements. For now, just try to think of segments as the logical controller of a table (or index, etc.). There are different segment types, but don't worry about those now. EXTENTS If segments are the traffic cops, you can think of extents as parking lot attendants and data blocks as actual parking spaces. So, the traffic cops tell you which parking lot to use and the parking lot attendant tells you where to park in that lot.
(SEGMENT-> EXTENT-> BLOCK) Now the job of your parking attendant (extents) is to make it easy to park your car. If the parking lot attendant were a workaholic and liked to operate in crisis mode, he would just tell each car that entered "You park here, you park way over there, you go up to the next level." But a good parking attendant will want to make life easier for himself and his customers. If you were part of a caravan, you would probably want to park next to other members of your caravan. And the parking attendant would want to make his life easier by handling cars in groups (like maybe 10 at a time) and say "you next ten cars park in slot 1-10" ( parking attendants don't like to start with zero). This improves allocation. You see, as Oracle puts it "an extent is a specific number of contiguous data blocks allocated for storing a specific type of information." So instead of allocating space one data block at a time, extents let let you allocate a group of datablocks at a time. Though those blocks get allocated (reserved) for a segment (hence a tablespace), they don't actually contain data until you INSERT data into your table (segment). If you use DICTIONARY MANAGED TABLESPACES (DMT's) you have lots of options on what your intial extent size is (basically how many blocks), what you want your next extent size to be (when you outgrow your current allocation and need room for new rows) and if you want your your new extents to grow in a certain pattern (using PCTINCREASE). This does not change the size of extents you've already allocated, just new ones (unless you rebuild using MOVE or something). So, if you had INTIAL =32k, NEXT=64k, PCTINCREASE=100 then your first extent would be 32k in size, your second would be 64k, your third would be 128k, and your fourth would be 256k, and so on and so on. As you can see, you would have many different extents of different sizes which can lead to fragmentation and even the possibility that you won't have enough room to even create your next extent (because the size requirement keeps doublin even if you just wanted to add a single row). Wow, that sounds like it can cause problems, doesn't it? Well, it often does. So why does Oracle allow this to happen? Because Oracle is an incredibly flexible piece of software. Oracle doesn't give you all these options because the developers (developers wrote it, not DBA's) couldn't figure out what would work best, they did it because they wanted to give you the ability to build your database the way you wanted to based on your needs. Unfortunately, too many people don't bother to learn the basics or analyze the implications, they just follow a script they saw somewhere or make some guesses
without reviewing them for impact. Properly used, this flexibility is incredibly powerful. But DMT's have inherent issues that DBA's cannot address. This is basically because of the serial nature in which things HAVE to be allocated. Oracle has relieved most of this allocation contention by moving allocation management from the entire database to the individual datafiles (where the lines could be a lot shorter). For that and other reasons, almost everybody now uses LOCALLY MANAGED TABLESPACES (LMT's). --------------------------To summarize, table data management is the responsibility of the table's SEGMENT (logical) which employs EXTENTS (logical) to marshall BLOCKS(physical). So, how do segements and extents relate to tablespaces and datafiles? We know that datafiles and blocks are physical constructs while the other three are logical constructs. Datafiles consist of one or more blocks (I've never seen a single block datafile, but I guess it's possible). Tablespaces contain one or more segments (a tablespace can contain many tables and indexes). If you've studied "create tablespace" and "create table" statements, you've seen that each allows for the storage clause (INITIAL, NEXT, MINEXTENTS, MAXEXTENTS, and PCTINCREASE, etc.). This is so you can setup defaults for all tables at the tablespace level or customize storage at the table level. Extents always belong directly to the segment, but since segments reside in a tablespace, you have this option to set up stoarge parameters at the tablespace level if you want. Ok, hopefully this was understandable because we are about to get to the last piece that will enable us to really start understanding most of what you'll ever need to know about how Oracle works (data blocks). Sounds too good to be true? Stay tuned. Of course, we will come back to these topics again, but by then you'll be a master.
Data Blocks Part 1

0 If you could only pick one thing in Oracle to know well, you should pick the DATA BLOCK. It is the key to everything worth doing in Oracle. If you can understand the data block, everything else in the Oracle RDBMS is easy. EVERYTHING. Another name for an Oracle data block is a "logical block", but you won't hear many people call it that and I'm glad. Because for now I want you to consider it a physical construct in our layers of logical and physical pieces. That's because data blocks impact/influence/determine storage, backup and recovery, SQL performance, memory usage, I/O, scalability, and probably anything else I haven't mentioned. And the cool thing is, data blocks are very easy to understand. --------------------------------------------A lot of my developer friends never get proper database instruction. As a result they are forced to guess about how an RDBMS like Oracle works. They know that databases store data and that a document server stores data. So it's easy for them I think to make the analogy that a database is just a fancy document storage tool with some fancy retrieval/manipulation tools. And they think, "So what? My app is cooler than that." So they don't invest too much time or attention in this "flat file repository." The ones that do have the time and interest sometimes are discouraged when they approach a DBA who is busy with his/her own work and may not even have a mastery of the basics themselves (people get mad when I say it, but if you can run a script, you can find someone who will hire you as a DBA). That's why you hear DBA's say "it's complicated" and developers say "Oracle sucks." Hopefully I can do a good enough job explaining these basics that you can avoid these retorts. --------------------------------------------
WHAT IF ORACLE DIDN'T USE DATA BLOCKS? If Oracle didn't use data blocks, then data would be stored in one big file and developers would be right to be unimpressed. That would mean that the database
would just be a big sequential file and it would be a mess to get the data in and out, you'd have to scan the whole thing everytime you wanted the data. At the other extreme, you could save every row of data as a seperate file. Then you'd...well, it's not hard to see how unworkable that would be. In short, without data blocks, Oracle really would suck. Data blocks are the way Oracle groups things like rows into managable sets. How many rows you can fit into a data block depends on how big your row is and how big you make your data block. And the beautiful thing is, you never have to deal with data blocks directly. You just have to understand them so you can design how you want to use them. HOW BIG IS A DATA BLOCK? As big as you tell it to be (kinda). Oracle doesn't force you into an arbitrary data block size. You determine what your needs are and pick an appropriate size. The most common sizes I see for data blocks in regular (OLTP) databases are 4k or 8k. ------------------------------------------------------NOTE: check the block size of your OS before picking a block size, you want the data block size to be some multple of the OS block size (1...64?) to avoid unnessecary IO. OS's read from disks in blocks as well and you don't want an uneven relationship between the OS and Oracle block sizes. Now Oracle even allows you have multiple data block sizes within the same database so you can really customize it to your needs. -------------------------------------------------------The basic idea in choosing a size is that if you do a lot of single row operations --you tend to like a smaller block size; and if you're doing primarily bulk operations --you tend to like a larger block size. That's mainly because of performance reasons, but we'll get to those when we start discussing SQL. For now let's just say that for a transactional database we'll use a 4k block size and for a reporting database we'll use a 16k block size. Now let's forget about the reporting database and just focus on the transactional database for this discussion. I have data blocks that are 4k in size. That means if I have rows that are 1k each (not a small row, but I'm keeping the math easy), I can squeeze 4 rows into each of those data blocks. Got it? Ok, now forget it, because you won't be able to get 4 1k rows into that block. That's because Oracle is not a flat file repository. And there are two extra parts of a data block that Oracle gives you to make things better: data block overhead and empty space.
FREE SPACE --The empty space is easy. The whole idea is that if you have a row of 1k which includes a text (VARCHAR2) column with the value "Dratz is a jerk", what if you want to update that record to a value of "Dratz is a genius, I hope he DOESN'T get eaten by a bear." Dratz is a jerk. Dratz is a genius, I hope he doesn't get eaten by a bear. See how much more space the second sentence takes up? If your data block were full, how could you update it? Oracle would let you, it would just move that row to another block that has room to fit it all in and leave a note behind telling the query engine where to find the real row (row migration). That means two reads instead of one to get that data. So it's a good idea to leave free space in every data block that you think has a chance of being updated to included more bytes. Read only data is not updated, so you don't have to waste free space with those. The free space also allows new rows to be added so you don't have to allocate new data blocks. When you delete rows, you are also freeing up space in a data block that be reused by new rows or expanding existing rows. OVERHEAD --The overhead contains some really neat pieces. I'm going to just mention the basic pieces today. In the next post I'll go into some more detail and explain how and why they are used. First, there are 2 header pieces. One is fixed in size for every data block and contains information about the data block, like what kind of block it is (data, etc.) and what the unique block address is. That may not seem very useful at this point, but we'll see how important they are later. The other header piece is variable in size and the important piece to remember here is ITL (Interested Transaction List). This is what allows concurrency. ----------------------------Concurrency really just means how many people can be doing things at the same time in the same place. It directly affects scalability. Many RDBMS products use a form of centralized "lock manager" to keep track of who's doing what to what. That is a valid solution, but as the number of concurrent users rises (not people just logged on to the database, but the people actively doing things) then a lock manager can and does become a point of contention. Oracle decentralizes almost all of this and let's the data blocks themselves deal with concurrent access (two or more people trying to manipulate rows in the same data block).
That is the reason why a well designed Oracle database can scale higher on less than any other database out there (no flamers, please). An interesting test I've performed (because my boss thought Oracle sucked and tried to prove it) is comparing Oracle to Brand X for a single user. Oracle was fine, but in numerous tests it lagged behind it's competitor. But, as I started to ramp up to dozens then hundreds of concurrent transactions (different sessions, etc.) I noticed something: the Oracle performance line stayed about the same, while Brand X started to fall behind more and more. ------------------------------------The ITL is controlled by the settings INITTRANS and MAXTRANS (just accept the defaults unless you really know what you're doing). As you can probably guess from those keywords, you set a starting number and then tell it how big it can grow (how many concurrent transactions you'll track before making people wait to get into the block). There is also a row directory that contains data about each row in the block. One interesting thing to know about this (small) component is that once space has been allocated in the row directory, you don't get it back when you just delete a row. It's interesting, as we'll see later, but not a thing to worry about or to feel you're being robbed over. All in all, the data block overhead is usually only about 100 bytes or more (let's round it up to 200 bytes and you can see that in a 4k block it's still a small percentage), so it shouldn't play too much of a role in your decision on a data block size. There are about 2 dozen other things in the overhead that I haven't mentioned, but they are as simple as the rest of it and we will address those later. I've mentioned the pieces I think you need to know for now. So, with this overhead, free space, and (wow) actual data, how is it all stored in the data block? SIMPLE BLOCK LAYOUT The data block header (fixed) is at the beginning of the block. A small tail is at the end of the block (the tail is for block consistency checking). Everything else (variable header and data rows) fits between the beginning and the end. Variable header info grows from the top down (just below the fixed header) if necessary and rows are inserted from the bottom up (just above the tail). Everything else is freespace. I will discuss free space and a thing called "freelists" in part 2 of the data block. Ok, in part 2, I will show some more examples of the data block and how the other structures we've talked about use and sometimes mimic the data block structure.
============================== Please give me some feedback on this post. I don't want to go too fast or too slow. Data blocks really are the key. Even though I'm simplifying them for now, there are other (still simple) parts to discuss. If I've already confused you, there's no point in going on. If you feel like you get it, you just don't know how it all fits together, we're ready to move on.
Data Blocks Part 2

Dratz May 3, 2006 | Comments (27) inShare1
0 Now it's time to see data blocks in action. For this example I am just going to consider a normal table with rows and columns and nothing more exotic (once you understand this, the various permutations (clusters, partitions, etc.) should be easy to understand). So, we've done step 1; we've issued a CREATE TABLE statement. Whether we accepted the default storage settings (by not entering a storage clause explicity) or we tweaked it for this specific table, a few things happen during the execution of that CREATE table statement: -Oracle creates a table segment in a tablespace and grabs some initial extents comprised of data blocks that sit physically in one or more datafiles. (reread the earlier posts if you don't get this part) -Those previously empty datablocks are preloaded with some small amount of overhead information (if you're still wondering why this overhead is good or even needed, don't get discouraged). -Oracle "registers" this segment in some data dictionary tables (adds rows) for database management purposes (it is a Relational Database Management System afterall). ----------------------------That last bullet might sound interesting. We have not INSERTed a single row of data yet, but there appears to already be rows loaded in the database. That is correct. One of the required tablespaces in Oracle is SYSTEM. The SYSTEM tablespace has a whole host of tables, views, indexes, and procedural objects that have nothing to do with your data, but everything to do with your metadata. You see, even the Oracle code uses the Oracle database as part of normal operations. This is great for two reasons: it makes the system very scalable; and it means that almost any DBA task can be done via simple SQL commands. -----------------------------So now we are ready to insert some data into our table (let's not worry about indexes right now). Let's say we have some data from an application or a flat file and we want to insert 1,000 rows into our new table. After some parsing (we'll also talk about SQL later), Oracle is ready to store the rows in data blocks. How does Oracle determine which data blocks to use? By looking for
freespace first. If Oracle can find some empty space in the segment, it will try and fill those empty spaces with the new data. If there is no freespace left, Oracle will try to add more extents to the segment and use those data blocks for storage. Because we just created the table, all of the allocated space (minus overhead) should be available. We know it's new and empty, but how does Oracle know? Well, Oracle could simply keep a list of all new tables that have never held rows, but that would only be of very limited use and could be a source of contention (if everyone inserting data had to check a global listing of empty tables, blah, blah, blah to see if their table was on the list). Instead, Oracle leverages the overhead in data blocks and does something similar at the segment level. That something is -- SEGMENT HEADERS. From a logical perspective, segment headers are similar in many ways to data block headers: they contain some management data about the segment. Segment headers contain a list of the extents in the segment (extents table), the FREE LISTS, and the HIGH WATER MARK. The segment header starts in the first block of the first extent in the segment. FREE LIST The freelist is simply a list of data blocks currently allocated to extents in the segment that have some free space in them for new rows. There are some interesting things to understand about freelists. Contention- instead of having everyone in the database wait in line to find free space in segments, freelists moves that contention down to the individual segment so only people manipulating data on that segment (table) have to contend with each other to find free space. If you have a segment (table) that you expect to have a lot of insert and update activity on, you can create multiple freelists to reduce that contention even more. To avoid waits, you can determine how many people will be inserting into that table AT THE EXACT SAME TIME (on the count of three, push your button) and create that many freelists. Unfortunately, many people adjust this number too high (thinking if two is good, a hundred is great) and end up wasting disk space (I'll try to remember to explain this when we start talking about design issues). A good DBA will monitor any freelist contention and adjust AS NECESSARY. Data block free space- in the first post on data blocks, I talked about free space inside data blocks. This free space is used to expand rows (via update) and for new rows (insert). In OLTP systems, almost every data block will have at least one lonely block that is not being used. Do we really want to see if we can squeeze a whole row into that single byte of space? No, it will never fit. We only want to try and put data where there is plenty of room. Fortunately, when we created the segment (CREATE TABLE), we told the segment (explicitly or by using defaults) how we wanted to report this free space inside of data blocks.
We did this using the keywords PCTFREE and PCTUSED. These are the keywords that determine if a table's individual data blocks shows up on the freelist group or not. These percentages (PCTxxxx) are just that, parts of 100. PCTUSED puts blocks on the freelist, PCTFREE takes a block off of the freelist (it is no longer accepting new inserts, only updates (DELETE's are always accepted and have no interest in checking freelists)). The default value for PCTUSED is 40(%) and PCTFREE is 10(%). That means if a block is only 39% used (<40%), it goes on the freelist as a candidate for new rows. It will stay on the freelist after new rows are inserted until the data block is 90% full (<10% free space remaining). Inserts increase PCTUSED(thereby decreasing PCTFREE), deletes decrease PCTUSED (thereby increasing PCTFREE), and updates can do either depending on if the update is increasing or decreasing the size of the row(s). Since our new segment currently holds no data, all of the data blocks in our INITIAL extent(s) are well below 40% used, so they are on our freelist. Any of those data blocks can be used to hold our 1,000 rows. Now, what if we had 1,000,000 empty blocks and were only going to use maybe 100 for this insert, would we really want to keep track of the freespace in all 1,000,000 blocks individually? I wouldn't. It could take longer to maintain and search for free space than just allocating a new extent. But then, like I hinted at above, I would be saving time, but possibly wasting disk space. So Oracle has another neat item that helps me out (not only here, but when doing things like FULL TABLE SCANS as well). That thing is... High Water Mark (HWM) - just like the name sounds, the HWM keeps track of how high the river of data has ever risen. While just counting the number and size of extents in a segment will tell us how much space has been allocated, the HWM will tell us how much of that space has ever been used (until to rebuild or truncate it). Hopefully, you can already guess at some of the benefits of knowing that. One of those benefits is in controlling the freelists. The freelist only tracks blocks that are below the HWM. That keeps things very manageable and reduces the chances of wasting space and resources (like by having 1,000,000 blocks with a single row each as opposed to having fewer, more densly packed data blocks). Let's say we have 2 extents each with 10 data blocks that can each hold 50 of our 1,000 rows. Let's also keep or PCTUSED=40 and our PCTFREE=10 So our block count=20. All of them are empty.
Because we've never inserted rows into this segment, our HWM is 0. When we insert the first row, the freelist will say "I don't have anything free, let me try to add some empty blocks." The freelist sees that there is a block already allocated to the segment (from one of our 2 extents) that is empty, so it advances the HWM from 0 to 1 and grabs one of those blocks. The row is then inserted. When we insert the second row, the freelist says "I have a block waiting for data" and tells us where to stick the data (whether its being rude or not depends on how you treat the database --joke), decreases the PCTFREE in the block and increases the PCTUSED. This goes on for the next few dozen rows until we try to insert our 46th row (we know that physically we have enough space for 50 rows in each block). By then, something has happened. Our PCTUSED has grown to 90%, but we don't really care about that because the block was already on the freelist (any of our blocks with PCTUSED<40 will be put on the freelist). What we do care about is that the PCTFREE has gone from almost 100 down to 10 when we inserted our 45th row. At that point, the data block said "I'm stuffed, leave me alone." So it was removed from the freelist. Now the freelist says "I don't have anymore space below the HWM, I'd better see what's out there." It sees that there are still 19 data blocks that have been allocated but never used. So, it grabs the next one, puts it on it's list and advances the HWM to 2. If this sounds like overhead to you, you're right. If it sounds like unnecessary (or even undesirable overhead) you're wrong. When we start looking at the big picture, you'll learn to appreciate it. So now we continue to insert rows in the same fashion, filling up block after block in the first extent (no flamers!) until all the blocks are full (or at least they're 90% full) and the HWM has been moved up to 10. At this point, we have used all of the 10 blocks in our first extent. Now the freelist has to grab some data blocks from another extent (see the no flamers comment). The freelist checks with the segment and sees that we have more blocks allocated in our second extent that have free space. We'll continue to insert rows like we have been, moving up the HWM until it gets to 20. At this point, both of our extents' set of data blocks have been removed from the freelist because they are 90% full (only 10% free). Now when the freelist looks for allocated, but unused space, it won't find any, so it will extend the segment by allocating a new extent (with its set of 10 data blocks) and advance the HWN to start adding those newly allocated data blocks to the freelist. We can then insert the remaining 100 rows. -------------------------------------------In this example, we've seen how freelists are used to insert data into data blocks. We have seen our segment size grow to the sum of the extent sizes allocated and how the HWM is used in this process. We have seen how data blocks accept new rows and
when they reject new rows. What we haven't seen is the affect this has on the tablesapce and the datafile(s). Since a tablespace is similar in ways to a segment, we know that the tablespace has grown (logically) to equal the sum of the size of it segments and is (physically) still the sum of the size of its datafiles. But what has happened to the datafile(s) themselves? You'll remember, or can look it up, that in Chapter 1 of this series we said that a datafile had a predetermined size. When we create it, we told it what size to be. That datafile is allocated to the tablespace. Allocated but not used (for data). As we create segments, we create extents which consists of data blocks from datafiles owned by the tablespace that the segment was created in. Whew! Say that 10 times fast. As we allocate new extents for our segment, does the datafile get bigger? No, not necessarily. The allocation of that space goes from unused (free) to used as a segment needs it. So, both the tablespace and the datafile get filled up. When the datafiles get full (by allocation, not by inserting data) then you have to either add a new datafile or allow an existing one to grow. Tablespaces work in a similar fashion logically with extents. As you approach the MAXEXTENTS setting, you have to increase it or prevent the segment (table) from growing. ------------------------------------------------This is not the end of our discussion on data blocks. It is just the beginning. But it is very important that I do a good job of helping you understand the basic structure and role of data blocks (and their chain of custody up to the tablespace) before moving on to all of the the simple tasks in Oracle. That's part of why I simplified some of this without trying to mislead you. This is the way it works and you will discover any shortcuts I took as we continue. Understanding the data block is critical to mastering the Oracle database and all of the tasks that come with developing for and administering the Oracle database. If I've done a decent job with these 4 chapters, you are very close to seeing just how easy it is to use the full power of Oracle and correct yourself of any notion that "Oracle sucks." I still don't have any volunteers to proof-read this, so please leave feedback if it was helpful or if it was confusing. I don't want to move on to transactions, backups, tuning, etc. if we're not comfortable with the basics of data blocks.
Basic Memory Structures

Dratz May 10, 2006 | Comments (3) inShare
0 If we had to rely on pure disk I/O for everything we did in Oracle, not only would our hard disks stay very hot, but we wouldn't be able to scale or perform as smoothly as we'd like to.
Fortunately, Oracle includes some pretty robust memory structures to help us and our users out. The two major memory components in Oracle are the System Global Area (SGA) and the Program Global Area (PGA). I still hear them referred to as the Shared Global Area and the Private Global Area. Other than the word "global", that's not a bad way to think of them as we get started. The SGA holds the things that are shared amongst all users and the PGA holds things private to a (user) process. --------------------------------------There are two basic configuration options for connecting users to Oracle. The default uses "dedicated servers", meaning that each user session creates it's own server process. "Shared servers" (formerly- the Multi-Threaded Server option) creates a group of shared servers that are shared amongst user sessions. This works (and often works very well) because users are not constantly pushing buttons: they push a button, get a result, and think for a second or more about what they want to do next. That means there is is some (a lot) time that a server process is idle. In dedicated server mode, the process WAITS for the user to initiate another action. That process is happy to wait. In shared server mode, the shared server says "done, who's next?" and processes another request from another user. That keeps the process productivly active and not just sitting around. So, everybody should always run in shared server mode, right? No, not hardly. Can you mix and match? I'm not going to ruin the surprise; we'll discuss these servers in a post on "Connecting to Oracle" coming up soon. I'm only mentioning the concepts here because the rest of this post is going to assume a "dedicated server" configuration. Then you easily understand how it works with
shared servers later. ---------------------------So, let's just think out loud about some things we think might be useful for Oracle to put into memory. Bueller? Well, we've been talking a lot about data blocks, so it's probably an easy guess that data blocks are something Oracle would put in a "buffer cache". Ok, what else? Well, if we're going to be hitting a bunch of different tables and such, we might as well store data about those objects in memory instead of reading from disk just to get that information. Anything else? If users are going to be running essentially the same queries over and over, couldn't we parse them once and store the execution plan in memory and avoid some of the overhead of parsing queries? Yes, and? Even if we used shared (or pooled) resources, we'll have some temporary things that are all ours. Wouldn't those be good to store in memory? Right again. Is there anything else? Yes, plenty more, but we'll worry about those later. We've identified the basic elements we want to start with: - data blocks (containing 1 or more rows or data, etc.) - basic operational information (about tables, users, etc.) - shared items (like reusable queries, PL/SQL code, etc.) - private items (like my execution info, private storage, etc.) In the next post, we'll get into some details about the SGA and which of these elements belong there and then a post on the PGA. One last memory concept I want to introduce here is the Least Recently Used (LRU) algorithm. I'll have a short post on it after the PGA, but it's worth mentioning here. Some people think "the more memory, the better." While generically that's a true statement, adding enough ram to create an Oracle "ram disk" is usually not practical or even necessarily efficient. So, what happens when you fill up all available memory and need to put something
else into memory? Oracle ages old things out of memory until it frees enough space for the new things. The LRU is the mechanism that handles all of that. Like I said, I'll discuss it in some detail soon. I just don't want those who think ahead to start creating problems that might not exist
The SGA
0 Please see my note at the bottom of this post about the new IntelliTXT feature ITtoolbox is testing (those green links with double underlines). ---------------------------------------------------------
The System Global Area (SGA) is powerful set of memory components that help you make Oracle very performant and scalable. ---------------------------------------------------------I say that it helps you because it won't necessarily keep you from shooting yourself in the foot if you really want to. The most important component of any database is the DESIGN. If you design your database (logical and physical models) poorly, no amount of database configuration will make up for it. If you've ever uttered the phrase "performance is a hardware issue", you're probably not going to get a lot out of this series. Performance is always a design issue and hardware is only one piece of the puzzle. ----------------------------------------------------------
The most obvious and immediate benefit the SGA provides is keeping you most active data blocks (rows) in memory. Even if that's all it did, it would probably be worth it. If you don't believe me, do this: - run some commonly used queries (SELECTs) against your database - shut down your database and start it back up - run those same queries and see how much slower they are That is because Oracle memory structures kinda work the way other parts of Oracle do (see how consistent and elegant Oracle is starting to look?). You'll (hopefully) remember that when we create a datafile, the space is allocated and the OS will not try to use that space for another file (even another Oracle datatfile), but the datafile has nothing in it. The OS says "yep, that's a 100 meg file", but Oracle says "I've got 100 megs of free space to put things in." The SGA works kinda the same way. When the Oracle "instance" is started, some memory is allocated from the OS for the exclusive use of Oracle. System memory is reduced, but Oracle says "I've got a bunch of RAM to put things in." Oracle immediately starts putting some important things in memory, but it does not
put rows of your data into memory.* It doesn't know which rows you want in memory until you start asking for them. As you start asking for rows to read or manipulate, Oracle puts them into memory block-by-block. Remember from an earlier post that the smallest unit Oracle really messes with is a data block. So even if you ask for a single row from a data block, Oracle will load that entire block into memory. This is a good thing, but you might think it's wasteful. On its own, it is not; however, you may see the effect picking a block size bigger or smaller than you need can have on memory usage (you're smart). So, you've just restarted your database and issued your first SELECT * FROM SOMETABLE WHERE SOMETHING=SOMETHING. After parsing your query, Oracle first checks to see if the blocks it needs to check are in memory. ----------------------------------------------------Make sure you understand that point. Oracle doesn't magically know where your row is, it uses your FROM clause to find the general area of where the row is (the table(s)) and the WHERE clause to limit those rows to the one(s) you're seeking. If you did not have an index and wanted one row out of a possible 10,000 rows from a table, Oracle has to read all of the datablocks into memory to find the one row you want. In that situation, your query wouldn't result in one row sitting in memory, but 10,000. It would simply give you back the single row you were looking for. More on this when we discuss the LRU and SQL. -----------------------------------------------------Database Buffer Cache The specific place in memory it searches is called the database buffer cache(dbc). Since we just started our database, Oracle won't find any of our rows in the dbc. So Oracle goes to disk to start sucking up blocks (rows) and pumping them into the dbc. Now, when the next user comes along with the exact same query we can avoid a couple of things: - generating a new execution plan (more on this in a bit) - reading these data blocks from disk You know, now, why you don't have to reread those blocks from disk--they're in the dbc. The execution plan, on the other hand, is not a block-based object. It is an instruction set that tells Oracle the best way Oracle thinks the data can be retrieved. -------------------------------------I'm intentionally over using "Oracle" as a uber process because I don't want to get into those details yet. We'll discuss all these individual processes soon enough. It can
just a little confusing to keep track of right now. --------------------------------------SHARED SQL AREA Since the execution plan doesn't really quite match the structure of a data block, we need some place else to put it in the SGA. That area, convienently enough, is called the shared SQL area. The shared SQL area stores things like the SQL statement and its execution plan. Hopefully other people will make the same request and we'll save a little time and a little memory by reusing the work we've already done. User Global Area Whether anyone else makes the same request or not, there's some information that will never be useful to anyone but us. This includes "bind information and runtime memory structures." Oracle needs a place in memory for this information and it uses a structure called the User Global Area (UGA) (there's that word "global" again, but you can see again that Oracle is persitently consistent). So, if the UGA holds session (or "user") specific information, where does it reside? Remember in the last post where I talked about connecting to Oracle via shared servers or dedicated servers? If you use shared servers (formerly MTS) to connect to Oracle, the UGA is stored in the SGA. That's because when you connect via shared servers, you are not locked into a specific (or "dedicated") shared server. One of your requests could be handled by one shared server and another of your requests could be handled by a different shared server. That's the whole point of using shared servers. BUT, your individual session still MUST have some private space that holds information only relevant to your specific session. By putting the UGA in the SGA, any shared server has access to it and can find your private information. If you use dedicated servers (one server per user), then the UGA is stored in the PGA and is the biggest chunk of the PGA. Ok, I will come back to the SGA because there is a lot more to go over, but let me quickly address the PGA before we do. -------------------------------------------------------OK, ITtoolbox is testing this new context-based ad placement called IntelliTXT. Just mouse over one of the highlighted words and you'll get a little pop-up message that you can click on to open a new window at the advertisers site. What do you think? Please leave feedback, because I know they are interested in hearing from you the readers more than from us the authors. I'm not going to stop writing just because I have green words popping up in my posts. I'd kinda like to see the same technology
for the links I do embed so you can get a preview of the link before going to it, but I doubt that will be available any time soon. I don't really like it or dislike it, but it's certainly better than that Microsoft ad that loads over the whole page anytime you accidentally mouse near it (I wonder if mentioning Mycrowsoft in that sentence will pull up that ad!) So, please give me some feedback on the content of my post and also leave a comment about the "mouse-up" ads. I'd be interested to hear what you think as much as ITtoolbox would. Specifically mention if it would make you less likely to read blogs here or if you find it better than pop-ups. I hated them at first (I fear change), but I'm already starting to tune them out. I'm going to check some of the other blogs here and see if I find ads that make me want to click them.
The PGA
0 Last time, we discussed some of the shared memory components in Oracle. This time we'll talk about Oracle's private parts (hey! get your mind out of the gutter). Specifically, let's talk about the Program Global Area (PGA). We are going to assume a dedicated server connection to the database (like last time). So, it's easy to see where we'll get some increases in performance and scalability by using the SGA properly. Can we get similar benefits from the PGA? Well, kinda. It depends on if your developers and designers know what they're doing or not. Design obviously affects shared resources as well, but shared resources don't keep copying the same mistake(s) for every server process! First let me just name a few pieces of the PGA: -Oracle code (the size of this is OS specific and the only thing you can do to affect it is change OS's) - a persistent area (once allocated sticks around for the life of that session or you CLOSE some things) - a runtime area (deallocated after the execution phase is complete)
Oracle Code Since there's nothing you can do about this piece other than migrating to another OS, all you can really do is be aware of it. BUT, if you investigate this in your OS-specific documentation, you could find that Oracle software code area on your OS is 2 megs per dedicated server. On another OS, you may find it is only 1 meg per dedicated server. So, depending on your OS, you could be doubling the startup size of your PGA. This is memory that is allocated from the OS for the private use of that PGA. ----------------------------------Two things to think about here if you need to scale to a high number of users: consider using SHARED SERVERS (formerly MTS) to greatly reduce the size (by reducing the number) of the PGA; or, if you prefer to/must use DEDICATED
SERVERS, install on an OS that has a smaller code area requirement. Just think about it, don't use it as a rule of thumb. ----------------------------------Persistent Area This part of the PGA does not go away until you tell it to. It holds things like bind variables. You DO know what a bind variable is, right? What's the difference between these two statements: select * from table where name='Dratz'; and select * from table where name= :b1; The first one is only "sharable*" when someone wants to limit the name to 'Dratz', the second one can be reused for any name by just plugging in the name at runtime. That's a bind variable. And it doesn't change the execution plan, so it can be reused by everyone. We'll get into this when we discuss SQL. -------------------------------------* Oracle is making it easier and easier to share cursors, but it's good to know what's really happening. -------------------------------------When does it go away? When you close the cursor. People run into this problem (and complain "cursors are bad") with EXPLICIT CURSORS a lot. That's because they leave a lot of cursors open that just keep taking up space. Just CLOSE them and you'll be fine (unless your design is so poor that you can't finish your transaction without opening dozens of cursors). --------------------------------------CURSORS AND THEIR CURSERS (I'm going to get sucked into a long SQL or PL/SQL discussion here. If you're not interested in cursors now, GOTO Runtime Area below) The word CURSOR has a very long history that I think is interesting, but "beyond the scope of this blog." For the last 30 or 40 years or so, databases have used the term 'cursor' and many people have been confused about them ever since. Here, I will explain just about everything you'll ever need to know about cursors. For one thing, most people (including me?) that try to explain cursors don't know what they're talking about and confuse personal preference with fact. To understand database cursors, I go back to the original Latin where the word cursor is used to "express the idea of someone or something that runs."
Now I'll tell you the big secret about cursors in Oracle: all SQL statements are cursors! Be careful who you tell that to, because some people will think you're an idiot for saying that, that you don't understand how complicated databases are, etc. Then they'll probably tell you they've got some kind of certification that proves they know what they're talking about. But the reality is that when a SQL statement is parsed, it is functioning as a cursor. Some people think that cursors are limited to only PL/SQL and many more people only think of cursors in their EXPLICIT incarnation. When I type in a DML statement and run it, I am running an IMPLICIT CURSOR. I have really no control over it other than what I put into the statement itself (like maybe a TO_DATE() or a GROUP BY). So, whatever I asked it to do, really acts on the whole result set. This is an example of a set-based operation. If I wanted more control over the set of rows I get back from a SELECT (like I want to evaluate a certain column or two in each row and decide what to do with that row based on the actual data values) then I could use EXPLICIT CURSORS. Explicit cursors are a type of row-based operation and give me a lot of control over how I handle the results I get back. Another difference between implicit and explicit cursors is how they are handled. For implicit cursors, Oracle handles them for you, meaning that the declare, open, fetch, and close all happen behind the scenes. For explicit cursors, Oracle expects you to tell it how to handle it (since you're taking explicit control of it). That means you have to explain how you want Oracle to open, fetch, and close the cursor. Not closing cursors is a common mistake that is easily fixed (if you just investigate). Is that everything you NEED to know about cursors? Yes and no. Yes because that's basically what they are; no because there's still lot's more to know to really make good design decisions. These other things will be covered in a later post and will show you how you don't really have to use the keywords OPEN,FETCH, CLOSE to control explicit cursors (what? but you said...) and go into more appllication focused issues like cursor variables (ref cursors) and cursor types like STATIC, FOWARD-ONLY, KEYSETDRIVEN, etc. You'll find people who swear by explicit cursors and people who swear they'll never use them. Avoid both types of people and be your own person. Keep reading these posts, understand how Oracle really works, then pick a solution that fits your needs. And don't let some clown tell you that "standards" force you to use one or the other. ++ blame/credit alexisl for getting me to add this sidebar to this post --------------------------------------
RUNTIME AREA This area stores things that are required during the execution phase of a transaction. In fact, as soon as you get to the EXECUTE phase, the first thing Oracle does is build a runtime area. The thing I want you to be aware of about the runtime area is what makes this piece big (or small). Say I want to get a list of names from my table in alphabetical order. I issue the following statement: SELECT NAME FROM MYTABLE ORDER BY NAME; Let's say it is the very first statement run after opening up the database (might as well refresh ourselves on earlier posts). What happens: 1. The statement is parsed (we haven't really gone over that yet) 2. We check the buffer cache (it's empty) 3. We read data blocks from disk and put them into the buffer cache (part of the SGA) 4. Our query process reads the data in the buffer cache and does what? a. copy the blocks from the SGA to our PGA? b. copy only the rows we asked for to our PGA? c. copy all of the rows to the PGA, but as rows, not blocks? d. copy nothing and just present the results to the client? That's a good homework question I'll let you ponder until we get to those topics, but since we're talking about the runtime area of the PGA, I have to mention sorting. Because I want my result set delivered alphabetically, something has to SORT the results and put the values in the right order. Right now, I'm not as interested in what does the sorting as much as I am in WHERE this sorting is done. I've read the rows from the blocks in the buffer cache and now I do a standard sort operation: I have a value; I get the next value; does the second value belong above or below the first value; get next value, repeat. Where I put these processed (and processing) rows is in the SORT AREA within the runtime area. Because it's in the runtime area, I know as soon as I'm done and get those rows delivered, I can reduce the sort area size (basically). I don't want to jump any further into sort areas now, there's more to consider than just the PGA (like when it goes from memory to disk, etc.). But it is important to know about because with older versions of Oracle, one of the ways to control the variable size of the PGA was to configure the settings for things like sort areas and hash join
areas. I'm not going into the details here, because they fit best elsewhere and because Oracle has moved away from those settings anyway to allow more flexible management of PGA resources (memory). See PGA_AGGREGATE_TARGET if you just can't wait. The most important things to know are the basics. So, in summary, the PGA has 3 basic pieces: 1. a fixed-size footprint that is OS specific 2. a "persistent" area that holds things like logon information and bind variables 3. a runtime area that holds notes about the current execution and provides space for things like sorting and hashing.
******************************************** I hope I haven't rushed through the SGA and PGA too quickly. I know there's a lot more things to discuss with each of them, but the basics are the most important pieces. Everything else will easily fall into place in context. The good news is that if you understand these first 7 posts, you know more than 1/2 of what you need to know about how Oracle really works. I'm going to start a couple of posts on Oracle processes next and then your training will be ready for the next level: becoming a junior DBA. You see, I'm tricking you. I'm teaching you all the hard stuff first. I'm just making it easy by focusing on the basics-- the stuff you really need to know. Then I'll show you how you can fill in the rest of the pieces as you need to. After I explain some important Oracle processes (little specific engines designed to do limited things very well), I will teach you the easiest thing to do with an Oracle database: back it up and recover it. I'll try to get there by the end of the week, trying to keep up my posts
Intro to Oracle Processes

0 So far we have discussed Oracle storage at different logical and physical layers and the basic memory structures. The last basic piece we need to in order to complete our "instance" is some code to actually do things. Oracle calls these bits of code PROCESSES. And it's a great name because that's what each of them does. I'm not going to explain how every process works. I'm going to introduce a few of them that I think are important to know now, but, more than that, I hope to give you an understanding of how processes work in Oracle so that when you come across some Oracle process you haven't heard about, you'll have a foundation you can use to help understand it's role. Some people find Oracle processes difficult to understand. Oracle documentation doesn't always make it easy. I'm going to try and break it down. I'm not ready to talk about the "listener" yet so I'm going to skip over explaining the difference between a connection and a session for now and just assume that we're connected to the database and are in a session. Besides, that's easy to do anyway. I'm going to focus on SERVER processes and BACKGROUND processes. Don't let the name "server process" confuse you. It is, indeed, a process run and managed by the server, but it's only job is to serve a user (using a dedicated connection) or a group of users (in a shared-server configuration). More about this in the next post. The other side are the "background processes." These are the small engines that keep Oracle running smoothly and efficiently. They do things like writing from the buffer cache to disk, cleaning up after users, and making sure the database is recoverable in the event of a failure. So, there is one process that handles my request(s) and a bunch of other ones that keep Oracle up and running for me and everyone else. See how simple it is? A process is just a little piece of (sophisticated) code that does a limited number of things.
So, when you start thinking of Oracle's "database software", don't think of it as one big clunky app, think of it as a confederation of small apps that all understand the basic architecture (just as you are starting to understand the basic architecure). I'll have a short post on server processes and a longer one on background processes, then you can print out all of these posts to-date and have the beginnings of a reference manual that might not answer all of your questions, but should give you the foundation to ask better questions and to understand the answers you get. Part 2 of this series will show you how to actually do things with what we've discussed.
Server Processes
1 A server process is one that handles user requests. When you type in a SQL statement, the server process handles the parsing and running of that SQL statement, reading datablocks from the SGA or datafiles (depending on whether or not the blocks (rows) have already been cached or not), and returning the result set to the calling application. Some people get confused when they hear this called a server process, because it sounds like it's really a client or user process. Well, it's not. It is a process run ON BEHALF of the client or user. And there's an excellent reason why these are kept as server processes. How do you connect to Oracle? Do you use SQL*Net (and a tnsnames.ora file)? Do you use JDBC drivers? Do you use something else? Do you use more than one way? If your user process were actually controlled and maintained by the client app or system, you would have at least two issues to deal with: performance and compatability.
PERFORMANCE
One of the many performance issues you'd face would be parsing and marshalling the raw result sets. If you've ever had to wait for your browser to parse and display a large XML document, you'll have an idea of this aspect. It would also require some minimum system requirements (RAM and CPU) that would be much higher than they are now. It would also be tying up network bandwidth moving al these chunks back and forth. With the process being run and managed on the server on behalf of the user, Oracle really functions in an SOA-type architecture. Plus you get to use things like the JDBC thin client very effectively.
COMPATIBILITY
The other issue you'd face if things were not as they are is that each client would have to be running a compatible client (thick) version. That immediately brings to mind
rollout nightmares where you can't upgrade the database because you can't upgrade all your clients. Of course, the reality is that you still have to have a compatible version even with the way things are, but those requirements are a lot looser. On my development laptop, I'm not even sure which Oracle client I have installed to use TOAD (I think it may still be an 8i version). But I have no problems connecting to Oracle databases versions 7.2 to 10g. Of course, it's still a good idea to use the version designed for your version of the database. I once ran into a problem using an Oracle7 client hitting a 10g database in shared-server mode. We got past it, but the "easy" solution would have been to upgrade 1,000 clients in the field, something we could not do in a timely manner. And, again, we can take advantage of things like the JDBC thin client without worries. So that's part of why our "user" processes are actually server processes.
DEDICATED VS. SHARED

We've already mentioned the terms dedicated and shared before, so I'll just go over them again. DEDICATED This is the default connection method in Oracle. You don't have to configure anything to use it over shared servers. What happens with a dedicated server (meaning a server process dedicated to a single user) is that, when a connection is established, a process is spawned which immediately grabs memory from the OS to create a PGA for this session. This PGA holds all of the stuff the dedicated server process needs: - to "exist" at the OS level - to work with the Oracle database on the user's behalf - to communicate with the user's application Pretty simple, eh? That's why I use dedicated servers in my examples, just a little easier than... SHARED Shared servers do the same tasks that a dedicated server does, it just does some of it a little differently. For one thing, a user request does not directly communicate with the shared server. If it did, that would effectively make it a dedicated server process (saying this process is for this user). Instead, in this configuration (which you must configure), the user request is
connected to a DISPATCHER. This is where people tend to get confused when trying to set up shared servers. A dispatcher and a shared server are not the same thing and you manage them individually. A shared server is a independent process who knows no loyalty. They are workaholics who just want to stay busy. They don't care who they work for, they always say "done with that request, who else has a request?" Think of it like this... Say you have a lot of errands to do around town today and you need someone to drive you everywhere you need to go. You could hire a limo driver for the day (dedicated server) who will drive you from A to B, wait till your done with B and then drive you to C, wait for you there, etc... Or, you could call a taxi company (shared servers). The taxi dispatcher will let all the taxis know that you need a ride from point A to B. Some random taxi will pick you up at A and drop you off at B. When you're done at B and need a ride to C, the dispatcher will let all the taxis know that you need a ride from B to C. A random taxi (it may, coincidentally be the cab you had before) will pick you up at B and drive you to C. Etc., etc... So, since shared servers don't know or care who you are (no matter how much you tip them) and don't keep a personal diary of where you've been and what you're doing, they have to have access to some public records in order to know anything about you. That's why the user's session data is kept in the SGA instead of the PGA. The data that for that shared servers own survival is all that is kept in the shared server's PGA. So, using shared servers, not only do you expect to have fewer PGA's allocated, the one's that are allocated are much smaller that PGA's of dedicated servers.
SHOULD YOU USE DEDICATED OR SHARED SERVERS?

I don't know, what do you think? Hopefully you base all of your design and configuration decisions on answers to the question "why?" Dedicated servers are easy to use, you don't have to do anything. Shared servers take some thinking because you have to determine not only how many shared servers you want or might want to grow to, you have to do the same determination for dispatchers. Dedicated servers means the only waits I have are database related waits. Shared servers means I may have to wait in line for the next shared server to become available.
Dedicated servers means I may not be able to scale as high as I want without running out of RAM. Shared servers means I could double, triple, quadruple, (whatever) the number of users I can scale to without changing a single line of code in my app. Dedicated servers means I could have processes just sitting idle for long periods of time. Shared servers means a few long running queries could effectively lock out other users by making them wait. Decisions, decisions, decisons. See the hard part of mastering Oracle is not understanding it, but figuring out what you really want to do. Huh? What's that? Which configuration do I use? That's easy, I often use both. I have some hybrid systems (part OLTP, part reporting) I've inherited and I let the small, fast, OLTP users get the benefit of shared servers and let the DW users take advantage of dedicated servers. You can't get away from dedicated servers completely, you need them for certain DBA functionality. I could go on and on about these server processes, but I think it's pretty clear now (at least what we need to know for now). Please let me know if it's not. I'll start working on the "background processes" now.
Background Processes
0 When you go about designing and writing a new application, you typically identify the beggining, the ending, and everything in between. It used to be that you'd write apps that started at the beginning and went all the way to the end. That makes sense. Then people started to realize that if you have to keep doing similar steps over and over, why not just write that piece once and just reuse it. Then people realized if you were going to do that, why not just host the reusable stuff on a centralized server and only give users the pieces they needed and let them reuse everything else that everyone else was using anyway. It's funny if you look at a chart the history of software development. People started out writing very small pieces of code because they had very limited RAM and even less CPU (power). They were forced to small, efficient code. Because people kept wanting more and more (remember Bill Gates saying no one would ever need more than 1 meg of RAM?), those clever hardware people made bigger and faster machines. As a result, people started writing more and more lines of code (KLOC's sound familar to you old timers?) to do more things. Unfortunately, hardware advanced faster than programming techniques and people got used to bigger, faster machines. So they started coding a lot more junk because... well, because they could. The echoes of "performance is a hardware issue" bounced off the walls of IT shops wround the world. Then technology went and did a silly thing. It went on a diet. People wanted to take software with them, on the road, on a PDA, on a cellphone. And they wanted to share all their wonderful software with more and more people. So now you have IT shops struggling to find people who know not only how to write code, but to develop software (too very different things). You need programmers to figure out how to get things to work and you need developers to figure out how to get things to succeed. So what does any of this have to do with Oracle background processes? A lot, actually. Oracle is not perfect, but it has maintained a solid level of
imperfection throughout the years by focusing on the architecture. It's like building a house. You can have lot's of pretty rooms and colors, but if it's built on a weak foundation, it will eventually crumble to the ground. Oracle has locked on to a good foundation and continues to improve it. This foundation is the single biggest reason why Oracle is better than the rest (there, I said it, happy?). Oracle manages the RDBMS architecture through sets of individual processes. We've already discussed the ones that work directly for you, the user. Now we're going to talk about the ones that work for us, the users. They are called "background processes." They are the things you don't have to worry your pretty little head over as a user. But as a DBA, Developer, or Designer, you must understand them. If you don't, you won't be as effective as you should be. I'm only going to mention about half of the background processes, but they are the most important ones and, if you understand them, you can teach yourself the rest. These processes are - The Database Writer (DBWn) - The Log Writer (LGWR) - The Checkpointer (CKPT) - The Archiver (ARCn) - The System Monitor (SMON) - The Process Monitor (PMON)
DBWn The database writer (DBWn) should be easy for you to understand because we've already talked about everything it does. We know that the smallest unit Oracle really messes with is the data block. We remember that a data block is a collection of OS blocks that have been allocated from the disk for the exclusive use of a datafile (which is controlled/owned by a tablespace). We recall that our tables\segments are created within a tablespace by requesting a number of extents from the tablespace. We know that rows are stored inside the data blocks. We also know that there is a specific memory structure inside the SGA called the database buffer cache(dbc) that improves performance by letting us store data blocks there instead of having to read them from disk. What we haven't talked about is how those data blocks in the dbc ever get written back to disk.
Well, now we're going to talk about it. It's called the database writer process (DBWn) (the 'n' is often 0, but if you set up multiple writers, you could have DBW0, DBW1, DBW2, etc., that's why I call it DBWn instead of DBW0). The DBWn's only job in life is to write data blocks from memory to disk. That frees up the user's "server process" to just do what it wants and not waste time nor effort making sure whatever changes they make survive a database crash. So then, how, why, and when does DBWn write data blocks back to disk? The how is easy, it just does multiblock writes back to the datafiles. The why is easy to, because we ran out of (configured) memory and need to free up some space. Even the when is easy-- when we need to and when it's convienent (kinda). The way it all works together is that you run something like this: You can't find 'Dratz' in your table so you run a query like select * from mytable where name like 'Drat%'; The blocks you need are currently not in the dbc and there is no free buffer space left in the dbc that is readily available. The DBWn checks the LRU (described below) to find out which objects currently in the dbc have not been used for awhile. It writes those blocks back to disk and frees up the space in the dbc that they occupied. My server process then loads the blocks I need from disk into the dbc and finishes my query. I see that there was a data entry error so I issue an update statement to correct 'Drats' to 'Dratz'. That datablock that stores the row is marked as "dirty" when I change one of the values from 'Drats' to 'Dratz'. That block then sits there in the dbc for anyone else to use until one of two things happen: - somebody else needs space and my data block(s) are no longer "hot" - Oracle's recover controls kick in and issue a checkpoint(CKPT) ---------------------------------------Dirty blocks and the LRU All data you act on is in the database buffer cache (DBC). The dbc sits in memory.
When you modify a row, you are modifying the data block that holds that row (making it "dirty"). The modified block stays in memory for a period of time. That period of time is based on how interested you and other users are in that data block. To determine the level of interest, Oracle uses something called the Least Recently Used (LRU) algorithm. All the LRU does is identify how hot or cold a data block is. Hot=a lot of people using it; cold=hardly anybody, if anyone, is using it at all. Cold buffer blocks are not flushed back to disk as soon as they become cold. They are just the first buffer blocks flushed back to disk whenever something IS flushed back to disk. There are some tricks Oracle pulls to help you out with the LRU. For example, if you do a full table scan (not using an index), Oracle assumes you know that you are probably not using a well tuned query. So, instead on letting all of the data blocks from that table (segment) sit in memory while more useful things are flushed out, Oracle will mark the blocks from the table scan as being the least recently used blocks. It's like cutting in line to be evicted! However, sometimes you don't want those blocks flushed too quickly. Oracle may think it's a poorly tuned query, but you're sure you know what you're doing. ALTER TABLE...CACHE To prevent Oracle from putting your table scan blocks at the front of the eviction line (LRU), you can use the very misunderstood cache option in your CREATE or ALTER table statement. To be fair, it's easy to misunderstand that option because when you ask for something to be cached, you expect it to be stuck in memory, that's the way caching works. But, lucky you, you understand how Oracle works and immediately realize "hey, if I cached a really big table all in memory, there wouldn't be room for anything else!" And you'd be right. The cache option does not load your entire table into memory and let it sit there. In fact, the cache option doesn't load a single data block of that table into memory. What the cache option is used for is to prevent the LRU from putting those blocks at the front of the eviction line. As a result, when a query requests rows from data blocks in cached tables (segments), those data blocks go to the "hot" end of the line, not the "cold" end of the line. Just like blocks from indexed queries. Congratulations, you can now explain something that a lot of candidates I interview can't. ---------------------------------------
Ok, so I've got plenty of space left in my dbc for new buffers, so why would DBWn
be writing blocks back to disk? BECAUSE WHEN YOU KICK OUT THE POWER CORD ON YOUR SERVER, YOU LOSE EVERYTHING THAT WAS IN MEMORY. Unless you're willing to lose everything that's been modified in your database, you'll want that data stored safely on disk. ** Just because DBWn writes your dirty blocks to disk doesn't mean that those blocks are removed from the dbc. That only happens when it needs space. In effect, it's just backing up what is in memory. The rest of the background processes we will discuss all work (constantly and in the background) to either keep your system from crashing or making it very easy to recover if it does. Log Writer (LGWR) So far, I think the only files I've mention have been datafiles. Those are the ones that hold your data. But they are not the only files needed by Oracle. One of the most important set of files (all of them are needed, but these are neat) are the REDO LOG FILES. REDO LOGS The redo logs record EVERY change made in the database. This includes create, drop, alter, update, insert, and delete statements. It's like a VCR for your database (or, if you're like me, a Tivo). NOTHING happens in your database that the redo logs don't know about. In fact, nothing happens in your database UNTIL the redo logs know about it. That's so if you do kick out the power cord on your database, you can rewind your redo logs to a starting point and recover everything that was "in memory." Sound like a bottle neck? It can be if you don't configure it for your usage. Does it sound a little silly do have a copy of a transaction in the redo log if you're going to put it on disk anyway? A lot of people think so and are constantly trying to figure out a way not to generate redo. But they're wrong. The redo logs are your best defense against corruption and media failure. Such a good defense that it is the one thing in the Oracle RDBMS that Oracle will not let you get around. If you think you're generating too much redo, fix your design! Back to the bottle neck question. Yes, if it were a direct-to-disk write, it could really be a sore point. So the LGWR uses a redo log buffer where server proccesses write their transactions to. The LGWR is scheduled to actually write to disk whenever: - you issue a COMMIT (fast commit mechanism) - 3 seconds have gone by without a write - when the redo log buffer is 1/3 full (likes to keep a lot of freespace so it doesn't prevent you from working) - the DBWn wants to write a modified block to disk that hasn't been written to a redo log (Oracle's write-ahead protocal) Ok, I know that's a lot to digest so I'll go over it a little more.
The REDO LOG is the authoritative voice of database changes in a live database. I will go into more detail about redo logs when I discuss back-ups. For now, just understand that all changes are recorded in the redo logs. When you commit a transaction, it is immediately written to the redo log file(s). Those blocks that you commited can stay in the dbc for some time before they are eventually written to disk, but the commit (and the system change number (SCN) it generates at commit) that is recorded in the redo log(s) is your receipt that your transaction has officially been registered and is as good as gold. What if 4 people commit at almost the same second? LGWR will immediately write the first transaction to make it into the redo log buffer and the other 3 have to wait. However, as soon as that first write is complete, LGWR will write the remaining 3 transaction in a single write. This method requires a lot less I/O in a heavily used system than writing every transaction individually. When it detects a commit, it checks for all commits already in the redo log buffer and writes them all. CHECKPOINT (CKPT) The CKPT does one thing, but to understand why it does that, we need to discuss a couple of things. One will be discussed in the backup post. The thing we can discuss here is a checkpoint. The CKPT does not initiate a check point, it just responds to it. One of the things that does initiate a checkpoint is a log switch. When dealing with redo logs, you could just have one really big one (if Oracle would let you) or several smaller ones. Remember that I mentioned that commiting a transaction does not automatically cause DBWn to write the blocks from the dbc to disk (that would cost too much I/O). I also mentioned that the redo log is the authoritative voice on what is a commited transaction or not. Now, during a recovery situation (like kicking out the power cord), Oracle must read transactions from the redo log and apply them to the database (datafiles) in order to fully recover the database. If you just had a single 2 terabyte redo log, then whenever you had to recover, you'd have to wait for Oracle to process that big file before you could open the database back up. So, you try and decrease the time it takes to recover by keeping those redo logs as small as you need. When you fill up all of the space in one redo log, it's time to switch to another one. That's a log switch. And that forces a checkpoint. A checkpoint tells CKPT to update all of the datafiles (actually just the datafile headers, remember those?) and let them know what the current state of the database is. This is important in a media recovery scenario (which we'll discuss soon) because it determines which datafiles need to be restored from a backup. Like I said, there's a lot more to go into, but that's for the backup post. For now, just understand that CKPT is the process that keeps the datafiles consistent.
ARCHIVER (ARCn) There are two basic modes in which to run your Oracle database: ARCHIVELOG or NOARCHIVELOG mode. This archiving is in reference to your redo logs (called the online redo logs). If you are not archiving your online redo logs (meaning you are running in NOARCHIVELOG mode), you might as well stop reading these post, sorry for wasting your time. You've obviously decided your production database is not important enough to be able to recover and that you don't mind losing transactions. If you are (sensibly) using ARCHIVELOG mode, you will be using the archiver process (ARCn). The ARCn is very easy to understand: it copies your redo log to an archived version of itself and marks that online log as ready for reuse. More on ARCn in the backup post. SYSTEM MONITOR (SMON) SMON is like the white blood cells of the database. It is always running around making sure things are ok, cleaning up things, and generally lies in the background looking for trouble makers. If another process finds a problem, it can ask SMON to investigate. Where SMON really takes the spotlight is during a recovery situation. Guess which post will discuss this more? No, not the backup oost, the recovery post. PROCESS MONITOR (PMON) PMON is like a den mother (I know, I need to pick one analogy and stick with it) that goes around cleaning up after everyone (including users). If you use a Windows client against a UNIX database, it seems PMON is always gainfully employed. PMON does a lot of interesting things like cleaning up or restarting different processes (for the system or the user). But the most interesting thing (to me) that PMON can do is call "Game Over." If PMON decides that something bad has happened (critically bad, not a poorly tuned query) that jeapordizes the integrity of the database, PMON will shut down the database. And it doesn't ask anybody before it does. Then you're in a recovery situation. It may just be "instance recovery" where you simply restart the database and let the redo logs make everything right, or it may mean restoring a file from a backup.
If other processes detect an issue, they can ask PMON for help and analysis. If PMON can help, it will. If it can't it'll do a SHUTDOWN ABORT!
****************************** OK, I'm sorry this post was so long, but I wanted to present all of the background processes (I was going to discuss) together. The good news is that, outside of the Cost-Based Optimizer(CBO), you know all of the basics you need to know to really start mastering the Oracle RDBMS. Now all we have to do is start discussing the different things to do in Oracle (transactions, backups, etc.) and you'll know exactly how those things really work. The next two posts will be on backups and recovery because those are (seriously) the easiest things to do in Oracle. Just don't tell my clients!
Backups
0 Now that we've discussed the underlying architecture of Oracle in very easy to understand terms, we are ready to see how that architecture is actually used to perform various tasks. That's why I've changed the title of the part to How Oracle Works. Ok, the single most important aspect of a DBA's job is to make sure that we don't lose data. Performance, data integrity, etc. are critical as well, but, if you lose years of data, you could be out of business (I had a client recently that almost faced that). It is perfectly natural for a DBA, even an experienced one, to have a minor heart attack whenever someone shouts from down the hall "We have to recover the database!" That's because everything up to that point has been easy money. Now it's time to prove your worth and save the company from impending doom. If you can't recover the database, you should be fired, and possibly sued. You'll certainly have to move to find work. Everyone will know you're a fraud. That's the gut reaction I have every time I realize I have to recover the database. I'm just being honest with you. But, as I learned more and more about how Oracle really works and stopped hoping that all the "how-to" manuals would walk me thru it, those panic attacks have reduced in duration from hours to less than 5 seconds. I don't think that feeling will ever leave me. I hope it doesn't. Because as soon as I feel no fear about my ability to recover my database, I'm going to start making mistakes. My fear keeps me sharp. Let me say, however, that if you were ever on site with me during a recovery (even early in my career), you'd never think I had a single concern. Because it is also my responsibility to reassure everyone that everything will be ok. Having said all that, let me now tell you how easy it is to do various backups in Oracle. LOGICAL BACKUPS A logical backup basically copies rows using SELECT statements on whatever tables you want to back up. Oracle's EXPORT/IMPORT is the most commonly used and probably best tool ever for doing a logical backup. It's the method I and almost
everyone uses. The thing to remember about logical backups is that you are not copying the datafiles, you are extracting the rows from the datablocks in those datafiles (via SQL) and putting them in another format (like a .dmp file or a .csv). The benefits of logical backups include the ability to back up specific objects you want (like 2 important tables or something. whatever that means). The biggest downside of a logical backup is that you can only recover data from the time of that backup and NOTHING after it. That's because a logical backup breaks the Oracle architecture and there is no way to reconcile it. It "breaks" it partly to make it transportable. You can import a logical backup to a different version of Oracle or to a different OS. Before 10g, you HAD to use export/import to migrate to a different OS. I rarely use export in a production environment, but I use it all the time in development. Those databases are smaller and I sometimes need to go back easily to a "known configuration." The only places I've found export to be routinely useful in production (other than migration) is with near-static tables. These are tables where something like a product list is batch-updated once a day. I can take an export after the batch process and know that I can recover that data any time up to the next batch process without doing a full database restore. Unless you're willing to lose a lot of transactions, a logical back-up is not a valid stand-alone backup strategy. In summary, a logical backup is essentially a binary (EXPORT .dmp) or txt (.csv) file that stores the results of a SELECT * FROM DATABASE. If someone adds/alters data after your logical backup, you won't get the changes. MEDIA BACKUP A media or physical backup is the method one usually associates with backups. This is where you make a copy of the datafiles and maybe put them on tape (I'll mention the RMAN option\method later). At the OS level, you use whatever COPY FILE commands you have available; it's kinda like copying a Word document from one directory to another. Before I talk about HOT and COLD backups, I guess it's time I mention the files you need to backup: - the datafiles - the control file(s) - the archive log files
Datafiles I think I've discussed datafiles well enough so far. CONTROL FILES The control file is the file that maps out all of the other files Oracle needs to start up and run the database. When you start the database, Oracle reads the control file to figure out what datafiles are associated with the instance and checks to see if they are current or not. They will not all be current if you had to restore one of them from tape or another disk location, hence, you'll need to perform recovery. The control file also keeps track of the location and status of your online redo logs and makes sure they are valid so the database can be opened. That's not all it contains, it also keeps track of checkpoint information, log history, tablespace info and a bunch of other things. It's a bit like a table of contents for your database. Oracle uses the checkpoint information in the control file to determine which redo logs have information that had not been written to datafiles before the database crashed or went down. A little bit more about checkpointing after we discuss... ARCHIVE LOGS As I mentioned in the post on background processes, the ARCn process makes a copy of an online redo log and tags it as an archive before marking that online redo log available for resue.
CHECKPOINTING Ok, let me see if I can now wrap up the important idea of checkpointing. I mentioned in my last post that one of the things that checkpointing did was keep the datafiles consistent. It does that by writing to the datafile headers and the control file and marking it with the latest SCN (remember that every transaction is assigned a system change number). The checkpoint is also the event that tells DBWn to start writing data blocks from the database buffer cache (dbc) to the proper datafiles. The reason all of this complicated mess exists is to keep your database responsive and scalable. Let me walk thru some database events and see if that helps clear up the roles these different pieces play. I run a select statement and get some data back. We know how that works.
I decide to update a row and so I issue my UPDATE statement and commit it. My row belongs to a data block. That data block (sitting in the dbc) records the changes my update made. My (dedicated or shared) server process sends a copy of my transaction to the log buffer. The LGWR will see my commit and write the transaction along with it's SCN to the current online redo log. If the database crashes at this time, then my changed data block will not have been saved to a datafile, it was still sitting in the dbc. The datafile has an older version of that data block. So when I recover my database, the following things happen: --Oracle checks my control file to identify all of the datafiles and online redo logs. It does a quick roll-call and sees that all of the datafiles are consistent, meaning they and the control file all agree on what the highest SCN was at the last checkpoint. --When the control file checks with the online redo logs, the redo logs say "we're here and we've got some transactions that need to be applied." --Oracle replies "Ok, the last checkpointed scn was 123. Redo Log 2, you should have 124 so let's start with you and start grabbing all of those transactions." --Oracle proceeds to go thru all of the online redo logs in order and recovers all the transactions and replays them so that every transaction a user committed is there just like before the database crashed. So, in summary, checkpointing makes recovery possible (and easy). Without it, you'd have to spend a lot of time figuring out which transactions need recovery. With it, you can very quickly and easily get the database back up and running. You can either leave checkpointing to occur at a log swith, or you can configure it to happen more frequently. Hopefully you have enough understanding now to make sense of the documentation if you feel you want to checkpoint more frequently than a log switch. LOG SWITCH So what is a log switch? Say I just have two redo logs. The reason I have to have at least two is for things like checkpointing and archiving. Say I do a lot of transactions and I want each of my redo logs to be 100megs in size. I start filling up my first redo log and when it gets to 100 megs, I have to change to my next log file. At this point, a checkpoint occurs and "dirty" blocks in the dbc are written to their datafiles. The ARCn process also starts making a copy of that redo log and gives it an archive name\number. Works like a charm. Where you run into problems is when you have so many dirty blocks to write to datafiles that DBWn cannot finish writing before your second log file fills up. When
your second log file fills up, Oracle will want to reuse your first log file (online redo logs are reusable). But, because DBWn is still in the process of writing blocks recorded in your first log file to disk, Oracle can't yet use it for new transactions. At this point, as far as the user is concerned, the database comes to a halt while DBWn tries to hurry and finish writing to disk. You'll see errors in your alert log like "checkpoint not complete". This is your clue that you need to rethink the number\size of your online redo logs and\or your checkpointing schedule. COLD BACKUPS Cold backups are backups done when the database has been closed. This is the only method of taking a consistent backup. A consistent backup is done after the database has been closed normally and means that all outstanding transactions have been applied and there is nothing left to be recorded in the datafiles. This clean shutdown is exactly why a SHUTDOWN NORMAL or even a SHUTDOWN IMMEDIATE/TRANSACTIONAL can take a while where a SHUTDOWN ABORT is instant; the shutdown has to wait until users finish their transactions (normal) or until you boot them off and rollback their outstanding transactions (immediate). I'm not a huge fan of cold backups in production, not only because of the time it leaves the database unavailable, but because I hate to lose all of that usage-based tuning Oracle has been working on since the last time I shutdown my database. (If you are crazy enough to run in NOARCHIVELOG MODE, then the only valid backup you can take is cold.) I find lots and lots of people who take cold backups simply because they don't really understand or feel comfortable with... HOT BACKUPS Hot backups are made while the database is up and running and USEABLE BY USERS (I'm amazed at how many DBA's tell me the database is unaccessible during hot backups). I think all of the confusion about hot backups come from a lack of understanding about how Oracle works. -----------------------------Let me go back to the Word doc example. Let's say two people were modifying the same document. I make some changes to one paragraph and save the document. The version of the document on disk records my changes, but not yours. I make a backup of the document. After my backup, you save your changes of the document. Let's assume for the sake of argument that Word reconciled our different changes and merged them into a single version of the document and saved it to disk. Then the disk crashes. When I recover the document from the backup, it only has my changes, not yours.
You work is lost forever. That's the way backing up shared documents work. But, we know that Oracle is not just a dumb flat-file repository. We know that there are structures in place to protect us from the above scenario. Those structures are basically the online (and archived) redo logs. And that's why inconsistent backups are nothing to be scared of. ----------------------------There's only one thing you have to remember about hot backups (if you're not using RMAN); YOU HAVE TO TELL ORACLE THAT YOU ARE TAKING A BACKUP. You do this by putting your tablespace(s) in backup mode. Two things happen when you tell Oracle to put a tablespace into backup mode: 1. the data file headers for each datafile in the tablespace stop recording SCN info from checkpoints 2. the entire block (not just the specific row(s) affected are recorded in the online redo log 1. Putting a tablespace into backup mode (via ALTER...BEGIN BACKUP) tells Oracle that users will be reading and writing from/to datafiles and that Oracle should avoid trying to keep up with whether or not the OS copy command copied an entire data block before it was rewritten (and having to copy it again, and again, and so on). So it doesn't keep up with checkpoint SCN's during backup. -------------------------------The reason this could happen is because, as we remember, a data block consists of multiple OS blocks. So the OS could read the first part of a data block, DBWn could write to that block, and then the OS could read the rest of the block. Remember that a data block has a header and a tail? In this situation the tail and header wouldn't match and we couldn't recover the data block. This is the problem people encounter when they think they are taking a hot backup but never put their tablespaces in backup mode. -------------------------------2. Since the data blocks (in those datafiles) could be changing during the backup of those datafiles(DBWn continues to write to them as normal), Oracle needs to be able to recreate entire data blocks. So, the FIRST time a change is made to a row that exists in a data block>segment>datafile>tablespace, that entire block is written to the online redo log. Any other time a change is made to that block during the backup, only the normal change record is written to the redo log. That's why you see a little extra redo generated during hot backups. Does all that make sense? During a hot backup, Oracle continues to run as it normally does except CKPT doesn't update the data file headers of checkpoint SCNs (the highest system change number at the time of the checkpoint), and LGWR makes sure there is one whole copy of the data block in the redo log to make sure the header and
tail always match. It makes that data block copy the first time the data block is changed and then every other transaction is recorded as normal. So, datafile1 has an SCN of 124. I put its tablespace into backup mode and continue updating,inserting, and deleting data. CKPT runs and sets the new high SCN at 165. CKPT records this new SCN in the control file and every datafile NOT in backup mode. Why doesn't it? Because the OS is copying that file and we don't know or care where it is in that copy. If the database crashes, when we start it up, the control file will check all of the datafiles and see that datafile1 has a lower SCN than the rest. So, Oracle must go back to the redo logs to find the last SCN registered in datafile1 and replay all of the transactions against that datafile. The first thing it would do is rewrite the entire data block and then apply any additional changes found in the redo logs until the SCN advanced to the same number every other datafile had.
RMAN Recovery MANager is a tool for backing up and restoring your database that works differently from a regular OS backup. I will post about it next.
Ok, let me know if this help explains what a backup really is or if I still need to do some work on it. It's just a blog, so I'm open to comments and suggestions.
A note about RMAN

Dratz Jun 29, 2006 | Comments (13) inShare
0 Ok, thanks for being patient. Life has settled down a bit now. I promised a quick post about RMAN, Oracle Recovery MANager. Let me first say YOU DON'T HAVE TO USE RMAN TO TAKE HOT BACKUPS! I'm amazed at how many people still take cold backups because they don't understand RMAN and think it is the only proper way to take backups. In my opinion, RMAN wasn't even a very good tool when it first came out. It has now matured and is something that should be CONSIDERED by everyone. Don't forget the most important question...'WHY?" Well, RMAN is better than OS backups, right? Well, yes or no. If you are not using hot backups at all, then RMAN is useless anyway, right? So first start taking hot backups, then learn how to migrate to RMAN.
INCREMENTAL vs. FULL backups in RMAN Oracle has made it easy for people to not understand these backups in RMAN. If you're not using 10g and "true incrementals", just use full. Older versions of RMAN were not much faster because it had to do just as much reading as a full backup (read the whole datafile, backup anything that's changed). They were ONLY designed to save disk/tape speed and/or bandwidth to disk/tape. They certainly didn't speed up recovery. In 10g, they've FINALLY introduced a "change tracking file" and many other enhancements like incrementally Updated Backups. So, RMAN started out as a ham fisted tool that caused problems for a lot of people (but did help many who dedicated themselves to it), but has now become much more of an asset to the DBA. I use RMAN and I am glad I do. But I also use 10g. I still have problems recommending it to people on older versions. Read through the product docs and you can see what it offers (you'll notice RMAN
doesn't have to put your tablespaces in backup mode). It also seems to be a lot easier to use nowadays, but maybe I've just gotten used to it. Like anything in Oracle, there are lot's of powerful tools and features for you to exploit if you know WHY to. Don't be one of those poor saps that tries to use every feature or tool just hoping that it's the proper way of doing it. Analyze YOUR needs and use the pieces you'll need. You'll be more successful and get a lot more sleep.
Rollback/Undo
Dratz Jul 13, 2006 | Comments (12) inShare
1 How Oracle Works (13) - Rollback/Undo I probably should have gone over this already, but as long as I get it in before we talk about restoring the db, we'll be fine. -------------------------------------------------------The Oracle architecture has one of the dumbest components new database professionals can imagine: rollback segments (known as UNDO in newer versions). -for those keeping score, I'm being a little sarcastic People mistakenly think they first encounter rollback segments whenever they see ORA-01555 "snapshot too old" errors. That's just often the first time that you notice them. In reality, you use rollback segments all the time (rather, Oracle uses them on behalf of everyone). I'm not going to go into sizing or any other real configuration details because: this series is trying to avoid a "do this" approach; I'm just trying to make the concepts easier to understand; there have been a lot of improvements (over an already good system) and I don't want to try and cover 4 different versions of Oracle. Hopefully I will be able to explain the role of rollback/undo segments in simple terms. WHAT DOES ROLLBACK DO? READ CONSISTENCY If you run a simple query like "what time is it?" or "what was the weather like last week?", you proably won't have any issues: you're looking for a discreet value that that is either static, or not likely to be updated by anyone at the eaxact time your're looking for the answer. They are also very simple questions. If you had a more complicated question and were even fighting other users to get your question answered, you could have a question like "how much of a discount is a customer due based on bulk sales made over the last ninety days in which no other discount was applied?" If you don't have an ODS or something else prebuilt for that question, you'll have to go and ask parts of that question to several different sections of your database until you get enough pieces to answer your question. That's when you need a read-consistent view of the database. That basically means that your query has a START and a FINISH. So, you may need five tables to answer
your query and you may even need to make multiple trips to those tables. We've already walked thru a basic SELECT statement in earlier posts. So we know that a SELECT does not magically produce data; Oracle has to build that data by reading data blocks. Say the SQL engine identifies 2 data blocks that have to be read to get a part of your query answered (wow, at least it'd be fast, right?). So, your process reads the first block and while it is, some bozo comes in right after you start reading that first block and changes data in the second block before you get to it. What happens? Do you see the new data or not? ------------------------------------------To see, let's walk thru what happens when you UPDATE data (I'm going to skip over the DELETE/INSERT aspect of an UPDATE because I think that's fairly obvious and doesn't help us here). I have a table(segment) with 10 blocks in it. I UPDATE a row in the second block. Because I'm initiating a DML statement, Oracle needs a place to keep track of what I'm doing and how things looked before I started. It doesn't want to use the data block I'm actually modifying to track all that I'm doing because then there wouldn't be any room left for data. So, basically, Oracle uses a work area to hold data the way it looked before I made changes. And, whether I've explained it well or not, it's a good thing. Whenever I run a statement, my statement asks what the current SCN is. That SCN sticks with my statement, not because I will push that number anywhere, but because it represents a "point-in-time" that my statement started. Now (like for a select before I update) Oracle says that all of the data I see will be the way it looked when my statement started. Oracle calls this "Statement-level Read Consistency". That should be enough to answer my question above. If I start reading datablocks, then Oracle will only return one of two things, the data as it was when my SELECT started, or an error saying it can't find all the data as it was when my statement started (that's the "snapshot too old" error). What if I immediately rerun my select statement? Will I now see the new data in the second block? Could it cause problems if I did see the new data (sounds weird, huh?) ----------------------------------------------Whether you'd see the new data or not depends on your ISOLATION LEVEL. Oracle comes with 3 isolation levels: READ COMMITTED - the default. It means, if it was commited before your
statement/transaction started, you will see it READ ONLY - not an ANSI92 standard and SERIALIZABLE - the fun one. This is when you really need the entire transactions to depend on a "point-in-time". You can SELECT all day long in this mode, but updates could be blocked if the data changed underneath you since your transaction started. More about isolation levels in the future. For now, we're just using the default READ COMMITED. OK, so we have our select statement that we've run once, we're using the default isolation level, we know the data changed DURING our last select, so will we see the new data when we rerun the SELECT? Yes, as long as it was commited before we reran our select. See where this could cause problems in a large transaction where we run the same query over and over (for some reason)? More about these in, yes, a future post. Back to what happens when I run an UPDATE statement. Because I am (potentially) changing data in the data block, Oracle needs a backup of that data first in case I run into an error or simply decide to ROLLBACK because I changed my mind about making the change. So, as soon as I run an update statement, people are going to start using my rollback segment to get the last known (committed) version of that block. Once I commit my transaction, all NEW statements will see my change, currently running queries will see the old version which is in the rollback segment. ---------------------------------------Delayed block cleanout (DBC) Someone always suffers for the sake of consistency and concurrency, don't they(assuming DBC occurs)? In this case (after a commited update) it's the first select run after the commit. In most OLTP environments, this is not a problem (unless you get ORA-1555 errors). Ok, I have a data block. I initiate an UPDATE. Oracle updates the datablock, copies the original (last) version to a rollback segment and puts a pointer in the datablock to point to that rollback entry. Until I commit my UPDATE, other users looking for that datablock will find a little note saying: "Hey, this block has been changed, go look in the rollback segments for the version you need (to reflect what was here when your query started)." When I commit my UPDATE, then all the regular COMMIT actions kick off. As far as the data block and the rollback segment goes, when the rollback segment is notified that the commit is official (log writer gives it the thumbs up), it marks that transaction as committed. Then it goes about notifying all of the related datablocks that the
COMMIT is official. In our example, it was short and sweet, so there are probably no issues with this commit cleanout. However, if I were updating thousands or millions of rows, Oracle may not make the notification to each data block in time. By "in time" I mean maybe someone ran a SELECT in the same nanosecond that the rollback segment was notified of the commit. It is possible (happens a lot on some systems) that the query goes to the datablock only to find my note from above. So the query then jumps to the rollback segment hoping to find the version of data it needs only to be interupted. "Can you do me a favor?" the rollback segment asks the query. "That transaction just committed and I haven't had time to tell the datablock yet. Could you let Oracle know so it nobody else has to make this detour?" That is why you can see a query run slow right after a lot of updates have been run (doing some kinds of bulk loading) and that is also why you can see SELECT statements appear to generate REDO! That's one reason for DBC. It is also possible that the datablock was flushed to disk by DBWR to make room for some other datablocks from the time I started my update to the time I commited it). Then it really just follows the above example, but I mention it because it demonstrates that you could have uncommitted data written in a datafile and it's nothing to worry about. Oracle protects you from uncommited data as we will see in our next post: recovering the database. There are other reasons that could cause DBC, but Oracle keeps improving the commit mechanism. In older versions of Oracle, DBC was about the only way to do it. In newer versions of Oracle, the term UNDO has replaced ROLLBACK in terms of tablespaces, etc. There have been a ton of improvements, but under the hoods, they are still auto managed rollback segments (largely). That's also why you can still see ORA-1555 even on newer versions of Oracle. Lot's of stuff here, hope I made it a little easier.
Recovery
Dratz Jul 24, 2006 | Comments (13) inShare
0 When we talk here about recovering the database, we mean that one of two things has happened: your database has crashed because on an INSTANCE failure or because of a MEDIA failure. The type of recovery you do, depends on the reason for your crash. For simplicity's sake, we are going to assume this is a single-instance database (no RAC) and that we aren't using RMAN. INSTANCE RECOVERY Instance recovery is required when the database gets some critical error or someone just kicks out the power plug. Whatever the cause (logical or physical) we have to start up the database after an non-normal shutdown. An instance recovery is the easiest kind of recovery to do, because you don't have to do much. Just start the database and Oracle will recover it for you. When the Oracle instance starts, it recognizes that something bad happen since the last time the database was started. It checks to make sure it has the right controlfiles and datafiles and then will automatically apply redo logs for you. If you've ever run a SHUTDOWN ABORT and then restarted your database, you've done an instance recovery. MEDIA RECOVERY Media recovery is required when a datafile or controlfile has become lost or damaged. It's usually because of a disk failure of some kind that makes the file unavailable or unusable. With RMAN, you can also handle something called "Block media recovery" which is pretty cool, but beyond the scope of this post. The main differences between instance recovery and media recovery is that, well, we have to restore some file(s) and Oracle doesn't perform it automatically. If you just happened to be in your datacenter and saw the database crash and just tried to open it up and it didn't work, you'd need media recovery (and Oracle would tell you almost exactly what to do). So, just restore a properly backed up version of the file(s); make sure your archived logs are available; and RECOVER the database.
Your Oracle documentation can give you the syntax if you need it. THE TWO KINDS OF RECOVERY COMPLETE If you think a complete recovery means recovering everything up to the point of the crash, you're right. Everything is recovered. INCOMPLETE There are times, however, when you cannot or do not wish to completely recover the database. The most common reason why you wouldn't want to is if some user screwed up a bunch of data, or dropped a table, or something like that. Examples of when you can't do a complete recovery include: you lose ONLINE redo log(s); lose current controlfile; or you do not have all of the archived redo logs you need. If you do not have all of the archived logs bridging the time from when the recovered file was backed up to the time of the crash, you can only recover up to a point in time of the last consecutive archive log you have, it can't skip a log. This can be a very bad situation indeed and one you should protect yourself from. WHAT ACTUALLY HAPPENS DURING RECOVERY? The primary goal of recovery is to put the database in a consistent state. That protects data integrity. The next most important goal is to recover as much of the data as we want to recover. Let's say I have 3 tablespaces and every night I backup one of them (they're huge, small window, insert reason here). One the first night of the month, I back up TBS1. One the second night of the month, I back up TBS2. Now, on day 3, I have a database failure. Doing my normal crisis management, I see that I had a disk failure for TBS1 (all of TBS1 was on this single disk) and the hardrive melted. Fortunately, I keep the last 2 backups of each datafile on disk as well as the archive logs. TBS1 has 1 datafile: DF1. When Copying over DF1 from my last backup, I get a copy error on DF1. That file is bad and I won't be able to use it for recovery. So, I go to the backup I took before that (three days prior to my last backup) and copy the file successfully to my new, store-bought disk that I've configured to look like the one that melted. Now, because I move my archive logs after three days to another disk location (three days=complete database backup), I have to copy ALL of the archived logs back to my archive log directory so they will be available for recovery.
Now, if everything goes well, I am done copying stuff. I go into SQL*PLUS, mount my database, set autorecovery on, and recvoer database. Oracle will (try to) automatically recover my datafile (bring it up-to-date) by applying the archived and online redo logs. Oracle checks the SCN in the datafile header and sees that it is old (it has an SCN of 100 while my other datafiles have an SCN of 200). Oracle says "to go back to SCN 100, I need archivelog123." and it starts applying the log to my new datafile (the current datafiles don't need any changing). That makes sense, doesn't it? Sure, until you remember that even uncommitted transactions are written to both the redo logs and individual datfiles (which is a great thing while the database is up and running). So, in order to do this right, I'm going to have to keep track of committed and uncommitted transactions (ones in the logs and ones in the datafile when I took my backup). Therefore, I do actually apply changes to another set of datafiles: the ones that hold my rollback/undo segments. Sometimes people ask "Why do I have to backup my rollback segments/tablespaces?" This is why. This is called Cache Recovery and it's when Oracle is rolling forward redo log entries to (re) record every modification made in the datafile. When Cache Recovery is complete, it's time to rollback any of those changes that were not committed. This is called Transaction Recovery and it rolls back uncommitted changes that were either in the datafile at the time of the backup or introduced by applying the redo logs (remember my database crashed, so people could have had uncommitted transactions). Then I finally get a message (after a bunch of "Log applied")--"Media recovery complete" Then I can open my database (and take another backup!). Basically, that's it. See, backing up and restoring is easy! In reality, you do run into problems from time to time, but hopefully understanding what's really going on will make it easier for you to troubleshoot your issues.
What is SQL?
Dratz Aug 1, 2006 | Comments (20) inShare
0 What is SQL, how is it pronounced, and where did it come from? SQL is the acronym for Structured Query Language. It is a data retrieval and manipulation language. It is usually pronounced as "es-que-el" or "sequel". I pronounce it both ways and my ears don't even notice a difference anymore. SQL came from Dr. EF Codd in the late 60's and was embraced by scientists at IBM including RF Boyce (ever heard of Boyce-Codd Normal Form (BCNF)?). Though IBM (as it has done with most computer technology) was a heavy lifter making SQL a workable language, it was not the first to really exploit it (as with most of its computer technology). No, it was good old Oracle that released the first commercial product based on SQL. And, since Oracle's earliest customer was the U.S. Government, the popularity of SQL and Oracle was secured. Standard SQL is characterized by its ease of use and great power in basic operations. Because it does not really excel at advanced analytics, each vendor invariably creates its own extensions to ANSI SQL. Oracle is no exception. Where, in some technologies, being 100% ANSI or ISO compliant can be a good thing, it's usually not in the world of RDBMS's. This is largely because of the architectural differences of the various RDBMS platforms. Why use SQL? It is the most powerful and flexible tool available for dealing with databases. Why do some people hate SQL? Because they are (lazy?)...(stupid?)... it's hard to find a nice word, but I'll say they are just ignorant. Many developers think SQL is universal or at least should be. They understand perfectly that you can't take VB code and get a Java compiler to compile it, but they
don't understand that 2 databases can have even slight variations. As a result, they try to keep their DATABASE application code DATABASE agnostic and then say a lot of silly things like "Oracle sucks." We'll discuss re-educating developers (they want to succeed as much as you do) later. In the next few posts, we'll go over how to use SQL and how Oracle handles it.
What happens when you run a SQL statement?

0 This is what the Oracle SQL engine does in a nutshell: 1 create cursor Yes, a cursor exists for every statement run. If you need programmatic control you can use an explicit cursor that you define. Otherwise, an implicit cursor is built for you. 2 parse statement This is the big kahuna. A lot of complicated stuff is going on. The goal is to get out of this step as quickly as possible. The quickest way to get out is to submit a DML statement that has already been submitted (a soft parse); otherwise Oracle will have to go thru the whole parsing process (a hard parse). [more on soft vs. hard parsing in a future post] Here are the steps that happen during (hard) parsing: -Translate the statement and verify the syntax is valid -check the data dictionary and make sure the referenced objects exist -get parsing locks on all required objects so the definitions don't change -check and make sure the executing user has privileges to the objects -figure out the best way (execution plan) to perform the request -load all this into a shared SQL area so others won't have to get this far -(route distributed statements) [don't forget, more on parsing later] 3 describe results (Queries only) and 4 define output (Queries only) 3 & 4 are used to determine and define what the result set looks like (datatypes, etc) 5 bind variables This is where your variables (like department_id) get attached. PLEASE use bind variables (more later) 6 parallelize statement (optional)
If you want multiple processes to work on your statment, this is where Oracle splits it up for you (more...uh...later) 7 run This is when Oracle takes a big breath and executes your statement. If you are running a DELETE or UPDATE statement, those rows will be locked for data integrity until a commit or rollback. 8 fetch rows (Queries only) If you ran a SELECT, this is the step that gets your rows and sends them back to you (ordered if you requested them to be). 9 close cursor "The final stage of processing a SQL statement is closing the cursor."
The next couple of posts will discuss how SQL interacts with other pieces of Oracle
The basic SQL statement

0 Ok, I'm going to teach you how to write a sql statement and then I can get into all of those nice architectural pieces we've discussed. I know this is very, very basic, but I don't want to skip over anything. Writing SQL is like speaking English with a Bulgarian accent ("Who the Hell wants to go to Boise?" --inside joke). It was originally supposed to be an English query language and it still kinda is...as long as you don't speak English very well. There are 3 things Oracle needs to know from your statment in order to run: What Where (don't confuse with WHERE-clause) When (because the WHERE-clause goes here) like do something over there when it's 3:00 or delete from employees where term_date >sysdate-712; WHAT What do you want your statement to do? The basic options are SELECT ____ -- return some data [SELECT col1 FROM tab1;] If you want more than one column, just list them and seperate them with a comma. UPDATE -- change some data [UPDATE tab1 SET col1='a';] INSERT -- add some data [INSERT INTO tab1 (col1, col2, col4) VALUES ('b',1,sysdate);] DELETE --remove some data [DELETE FROM tab1;] *this would delete everything (from) WHERE Where to do WHAT from above. If there is more than one place (table) involved, just list them and seperate them with a comma.
See the use of tab1 above. WHEN This is an optional piece that contains the conditional logic you want to use. It is the WHERE-Clause. In most cases, you are making a big mistake by not using this optional piece. We will see how important the WHERE-Clause is soon, especially when we discuss indexes. In this statement delete from employees where term_date <sysdate-712; the where-clause says ONLY delete data older than two years. SUMMARY Yep, that's all you need to know about SQL. Not really, but that's 90% of it. I'm not trying to confuse you by putting the where-clause in the WHEN of "WHAT, WHERE, WHEN?", that's where it belongs. If you can remember "WHAT,WHERE,WHEN?", you'll be able to tackle advanced SQL statements with ease (like nesting, in-line views, correlated sub-queries, connectby etc.).
Excessive Redo Generation

0 Ok, by now you should have a fair understanding of the major architectural components used in Oracle and what they do. We've just discussed SQL because it is the major interface to communicate with Oracle. Now we are about ready to start talking about things like design and development. But, before we do, I think we should discuss REDO generation, why we have it, and what to do about it. There is a widespread belief that a phenomenon known as "excessive redo generation" exists and that it is an Oracle problem. Certainly the term has caught on. The theory seems to follow along the lines of "Oracle generates way too much redo, wasting space and time." People pull their hair out trying to solve this "problem". Some people even run in NOARCHIVELOG mode thinking that solves anything. By now, you should understand what the redo logs are and why they are critical. Now I'll tell you the fastest way to reduce redo generation for your database (not just Oracle): don't do anything in your database. The next option up is to make your database read only. The ugly option (the one least liked) is to (re)design your database properly. [ In a future post, we will get into design concepts that will help, things like "transaction lifecycles" and such.] This last option is unpopular for many reasons: 1. You pay a lot of money for Oracle and you want a magic button that bypasses the laws of physics and logic. 2. You already designed your database not accounting for how Oracle generates redo and you can't/don't want to redesign it. Ok, now I'll let you in on something. Excessive redo generation is not a myth, but Oracle's culpability is very limited.
People mention "excessive redo generation" as if it were an Oracle bug when in fact it is usually an ignorance of how any RDBMS works (and Oracle specifically) or it is an inefficient database/application database-application "database application" design. IGNORANCE I had a Senior VP of Technology who was in charge of servers, databases, and something else who called me into a meeting with his other direct reports and wanted to know from me why Oracle was generating excessive redo (they had searched the web to confirm their suspicions). I asked "which part of the redo logs are excessive?" "Well," he said, "that's what I want you to find out." As proof, he and the storage manager showed me a report they had just completed that showed our archive log generation* for one database was averaging 200GB a month. They laid that number down next to the number that was the sum of the size of the datafiles in that database: 250GB. He had himself a little worked up, but was doing a great job restraining himself (he was also trying to quit smoking). "How can we possibly, POSSIBLY be generating that much archive logging A MONTH when the whole database is just a little bit bigger?" Though my staff and I were administrating a lot of databases, I was very familar with the database in question. I don't have room (or the inclination to violate an NDA) here to explain the entire system, but this database was in many ways a workflow database (a huge one). The role it played in the company was that it tracked enterprise transactions that had started but not completed. Some of these transactions took several weeks to complete because of the human intervention required at each step. This system didn't do a whole lot with the transactions other than track and maintain some auditing data. When the entire set of source-system transactions finished and this enterprise transaction had completed, the enterprise transaction was moved to a DW type system. So, the whole nature of the data was that it came in (generating redo), got updated a lot (generating redo), then got deleted (generating redo). In essence, we had a completely new (and full) database every 35-40 days. After I explained that, they understood. They still wanted to find a way to limit the size of the database and I told them that was no problem, just tell me how many transactions to reject and I could constrain the size to whatever they wanted. POOR DESIGN This is a tough one to explain in a single post. In a nutshell, it means you should
streamline your application's database requests and you should limit the amount of unnecessary work your database does. These are not INIT.ORA type settings, they involve transactional modelling. This is one reason why I always talk to people about making stored procedure calls instead of embedding SQL in the application. If it takes less time to do 1 thing rather than doing 2 things, try to get away with just doing 1. Doing 1 thing twice counts as doing 2 things. This idea of doing only the necessary steps can, in some cases, be used with SQL options like NOLOGGING, but this is not a panacea. If you have a multiuser, production database-- redo is a good thing. When we get into design and development, I will try to remember to include redo implications. Feel free to ask if I don't. *(remember, that's the archived version of each redo log so we can reuse that "online" redo log)
Intro to Indexes
Dratz Nov 21, 2006 | Comments (32) inShare
2 "The database is slow, can you add an index?" Anyone who has ever worked on either side of a database has heard or asked this question many times. Before we get into the different types of indexes available, we need to discuss more important matters: why even use an index and, then, why not index everything? When building a new database application, in say Java or .Net, you can spend thousands of hours and millions of dollars building perfect application code only to have the entire system brought to its knees by a single simple SQL statement. All that hardware and all that software rendered useless by a SQL statement that won't perform or scale. So, assuming you have good application code, assuming your database design is solid, assuming performance is not a hardware issue, what simple thing can you do to save everything? You can stop and think. See, the whole idea of SQL tuning is NOT to figure out what index to create. It's to figure out the best way to handle the data. As it turns out, adding an index will quite often be the best way. But not always. Say I have a table that records a LOT of transactions during the day but isn't normally read (SELECT) during the day very much (I just load them into a data warehouse at the end of the day and truncate the table). The worst thing I could for performance would be to add an index. That's because updating, inserting into, and deleting from an index is an additional step: instead of making changes in one place (the table), now I have to make the changes in two places (the table and the index). ================= Oracle manages the index for you, there are no additional commands you need to use to keep the index sync'd with the table. ================= That might sound like a backwards way to explain indexing, but it holds true even
when you turn it towards queries (SELECT statements): it's faster to do less things than it is to do more things. That's the whole point of indexing in the first place. It's supposed to let you limit the number of data blocks you have to read to get your result set. A full table scan reads every row (meaning every data block in the table segment up to the high water mark). Well, if you only wanted one row out of that table, is it really worth reading the whole table to find it? Maybe, maybe not. What? Let me give you and extreme example. Let's say I have large data block size in my database and I have one table that I use a lot, but all of the rows exist in a single data block. Using an index to pull a row out of that table would require 1 read for the index and 1 read for the table. That's 2 blocks read to find that row instead of just doing a full table scan on that single-block table. So, clearly, in such a tiny table an index will cause more harm than good. But that logic isn't limited to single-block tables. That logic holds for a table of any size. Is adding an index going to help you or hurt you? That's what you have to figure out. We all know your goal: to make the query go faster. Your task is to determine if an index will help you or not. And that should be easy to do, because all you have to do is try it and test it. But it doesn't mean you just assume that will have nothing but a positive affect. Most of the time, I don't have to build the index and see if it will help, because I've already analyzed my data (and distribution), my table structures, my database configuration, and my transaction lifecycle so I already have a good idea if an index is called for or not. If I step into a performance issue on someone else's database, I run an explain plan to see what's really going on. Don't get me wrong, an index is one of the best features of an RDBMS and will solve so many of your performance and scalability issues. It can allow the SQL optimizer to avoid large chunks of data blocks and narrow down the candidate blocks very quickly taking a query that takes minutes to run to less than a second to run. It's magical to look at.
But I'm trying to teach you how to think like the Oracle database does, and an index is not a magic button. Next we will discuss different index types (yes, you not only have to know when to index, but what kind of index to use) and why they work different ways and solve different problems.Table of Contents for this series
B-tree indexes
Dratz Dec 7, 2006 | Comments (16)
0 Ok, now we're ready to talk about specific types of indexes. The first one we're going to talk about is the basic, default, great-for-almost-anything B-tree index. This is the type of index you get if you just use the basic "CREATE INDEX..." syntax. It is by far the most common type of index used and for good reason- it works very well. ----------------------------------------------When I teach database design and start talking about indexes, one of the first questions I get asked is "What's the max number of indexes I can put on a single table?" And I have to give an honest answer: I have no idea and I don't care. To me, as a proud and competent database designer, it's a dumb question. If you ever get a database error message that says "sorry, you already have too many indexes, try something else", then you should give back your super secret IT decoder ring. If you are indexing every column in every combination in hopes that that will fix everything that could ever go wrong, you are probably the kind of person who says "performance is a hardware issue". And you have more issues than an index is likely to fix. -----------------------------------------------HOW DO I KNOW IF I NEED AN INDEX? First: If your queries don't use a WHERE clause, you're wasting time on an index. IF you have a big table and IF you only want a few rows back based on a specific data value, then you could probably see a lot of benefit in using an index. For example: I have an excellent Oracle book that is about 1,000 pages long. If I want to do some reading on UTL_FILE for some clever code I want to write, there are a couple of approaches I can take: I can go thru the book, page by page, scouring the entire text for any reference to UTL_FILE; or I can flip to the back of the book and look in the INDEX under 'U' and
see that UTL_FILE is discussed on 3 specific pages. That's very handy. So handy, in fact, that databases do the exact same thing: they use indexes to make it easier to find something specific. Now, what if I wanted to look up the use of the word 'the' in this Oracle book? If I flipped to the index in this 1,000-page book and found an index entry for the word 'the', I would probably be drowning in page numbers that included the word 'the'-- it appears multiple times on every single page. An index on the word 'the' in this context is worse than useless. But what if I were doing some research on English grammar and was searching a book for references on the definite article 'the'? In that scenario, the index would point me to the section(s) of the book that discussed the use of the word 'the'. I use this example to demonstrate that there are no fixed rules that say certain types of data or certain data values must be indexed. It all depends on how you use it. Another thing you can use indexes for is to simply tell you what is indexed. Have you ever flipped to the end of a technical book to look at the index and see what kinds of things you can find? I use the index more than the table-of-contents. When I'm doing this, I don't really care what pages the entries can be found on, I just want to know what my options are. You can do the same kind of thing with your database. I don't like user entered text, it just screws up my data and makes it a lot harder to do any analysis. So I insist on the use of drop-down boxes everywhere I can. Don't type-- just click. In a well designed database, you can use a lot of your tables for this purpose without having to duplicate the data in "lookup tables." But I don't want the whole table for that, just a single column (or two). Say I have a little notification app that allows you to forward a report to someone in the department. You simply select their name from a drop-down box and hit 'Forward'. I already have a staff table with a bunch of detail information, I don't want to maintain another table with just their names in it. So I can create an index on the name column and the department column. This is called a COMPOSITE INDEX (because it is made up of more than one column). Since I only want this app to forward reports to the IT department, I populate my drop-down box with the following query: select name from staff where department='IT'; Now, everytime someone clicks on the drop down box (or whenever that data gets refreshed on the front-end), I know this query will return the names of people in the IT department and I know that the table (segment) will never be touched. What??? That's right. The table does not need to be read from at all because all of the information I cared about (asked about in my query) is contained in my composite index. Oracle won't read a single data block from the table. That can be a huge
performance boost or just a small one, test it and see if it helps you or not. -----------------------------Hmm, I really haven't discussed much of the technical details of B-tree indexes, have I? Well that's because step 1 is to determine if you need an index. If you do need an index, use a standard, default B-tree index unless you have need for one of the other types we'll discuss next. You should always have a well thought, specific reason to use any index type other than a B-tree. If you don't, do yourself (and your users) a favor and stick with the basic B-tree: it works very well. I will come back to B-tree indexes when I talk about function-based indexes, but first we will discuss bitmap indexes. Please leave comments here if you feel I haven't done justice to B-tree indexes. The most important things you should know is WHEN to use them and WHY NOT to use another index type. We'll discuss those other types next.
Bitmap indexes
Dratz Feb 20, 2007 | Comments (14) inShare
0 I've been thinking over the last couple of days of how best to try and explain bitmap indexes. It's one of those topics where I thought I had a good explanation, then thought it wasn't so good, then thought it was great, then wasn't sure. So I kinda started from scratch again and here it is. If you've worked with or read much about bitmap indexes in Oracle, you've heard a few typical warnings: -only use on low cardinality columns (like 'True'/'False' or 'Male'/'Female') -use them only in datawarehouses and try to use nothing else -do not use them on tables with heavy updates Well, I guees those aren't bad rules to keep in mind if you don't understand how bitmap indexes are built and maintained, but hopefully I can make them a little easier to understand and you can make decisions based on your good analysis instead of broad, misleading rumours. You can search the internet and find some real technical explanations of bitmaps and bitmap indexes, but I hope mine is easier to follow. In a simple B-Tree index, you really have each row of the table represented in the index one time (each) with basically 3 things: some logistical overhead stuff, a value from the column you are indexing, and the rowid where that row can be found in the table. So, you write a query and include "WHERE user='Dratz';" If 'user' were indexed with a B-tree index, Oracle would probably query the index, find out where my row is (out of millions of users), and find it for you very quickly. A bit map index is a little different. In a bitmap index, you really have a representation of every row for every possible value in your index. A bitmap index entry basically has 3 parts: some logistical overhead stuff, a value for one of the possible values in your index, and a list of bits (one for each row in your table). The bits go from the first row in your table to the last and have one of two possible values : 1 or 0. 1=match, 0=no match. A match means that the value you are looking for in this bitmap entry exists for that row.
So, if you really looked at the guts of a bitmap entry you'd see something like: (overhead):'Male':0110010101 Out of the 10 rows in my table, it appears that rows 2,3,6,8, and 10 represent 'Male' rows. Does that mean my other 5 rows represent 'Female'? No, it doesn't. While we know that the only options are 'Male' or 'Female' (we'll ignore unknown for now), this bitmap entry doesn't know or care. We could have a million other options than 'Male' or just one. This entry simply doesn't care about anything that isn't 'Male' (men!). So, if WE know that 'Female' is another option, what can you guess? Assuming the only two values are 'Male' or 'Female', there must be another bitmap entry that looks something like: (overhead):'Female':1001101010 From this information, we should be able to deduce that a bitmap index is comprised of 2 or more entries that are effectively independent of each other. When you issue a query against a bitmap index, the index sees what value you are looking for and uses the appropriate entry. Then that entry races thru its list of bits (meaning rows) to see if rows match (bit=1) or don't match (bit=0). Make sense? ***************************************** I know I haven't explained how, in a list of bits, Oracle equates a specific bit to an exact row, but let's just say for now that that mapping is handled in the index overhead. ***************************************** So, what if we decided we wanted to add that 'Unknown' value to our index and add one row that had that value of 'Unknown'. I have a product that I don't want to market to men so I decide to select every row from my little table where sex is either 'Female' or 'Unknown' (no bad jokes, Matt). I could write my query to include "WHERE sex='Female' OR sex='Unknown'; " (this isn't a SQL class). That represent two of the possible values in my bitmap index, doesn't it? That means two seperate entries, each with their own list of "rows as bits". In such a query on such an indexed column, Oracle would most likely try to take advantage of each entry. Because of the way bitmap entries store rows (as streams of bits), it's very easy to do such operations. Like:
Female OR Unknown 1 0 - MATCH 00 00 1 0 -MATCH 1 0 -MATCH 00 1 0 -MATCH 00 1 0 -MATCH 00 ------------------------------------------------------The new row I added 0 1 -MATCH
6 matches You see how easy it is for Oracle to compare these two bit streams? Would this same basic logic work if Oracle were comparing entries from two different bitmapped indexes (like sex and buys_online_flag)? Yes, it does. And if you can understand how the OR works, you can easily guss how AND operations work (except ANDs only work when comparing multiple indexes, see?). It's this joining of multiple bitmaps that provides the real power of bitmap indexes. Oracle just does some real simple math to find all of the matching rows ONCE, then figures out what those rows are and return the data to you. So, is joining 5 bitmap indexes more "efficient" than joining 2? Probably. That's one reason why bitmap indexes are so popular in datawarehousing: they're very helpful in ad hoc queries. So what about cardinality? Like anything in a relation database, it's all relative. I think Oracle still suggests a 1% rule. I've had success with even higher percentages, but the basic idea is the same: is it faster to use an index than not use one? If I had a table with 1,000,000,000 rows in it, would I worry about using a bitmap index on a column that had 50,000 possible values? That's a lot more than the 2 or 3 values used in most bitmap examples (and myths). The answer (for me) is- it depends. It depends on other factors not dicussed here (and that are often unique to each situation), but I would not look at 1,000,000,000/50,000 and automatically think a bitmap index wouldn't be useful. I think that's something people forget about cardinality -- it's all relative. Test it and see, that's the best advice I can give you. Ok, if bitmap indexes are so wonderful, why not forget about B-tree indexes altogether and just use bitmap indexes?
Well, first it would be disrespectful of the wonderful B-tree index that has done so much for us. The second is that it wouldn't be very smart. Why not? Because of things like UPDATE statements, the most costly of DML. To understand why, lets go back to how index entries are built. B-tree: overhead: value: rowid Bitmap: overhead: value: stream of bits If you've been reading this series (as opposed to finding it on Google or something), you'll know that indexes are stored on SEGMENTS (of type 'index'), and that segments exists as a series of EXTENTS, and that extents are created as a number of DATA BLOCKS. So, finally, we come back to the data block, our smallest unit of storage. Every bit (literally) of an index sits on a data block somewhere. Can you see how the structure of a B-tree index loosely resembles that of a data block? It has overhead like a block; a value like a row in a block; and a rowid that corresponds to a row's rowid in the table's data block. A bitmap index has the overhead and value, but instead of pointing to a single row, it lists EVERY ROW. So, given that a data block is of a fixed size, can you fit a 'bit' for every row in your table into a single data block before you run out of space in that data block? Hopefully your answer is: it depends. It depends on the size of your data block and the number of rows to be represented. But let's say that in very large tables, there are too many rows to fit them all (even as bits) into a single data block. What happens? Well, we simply use additional data blocks. This no problem for a bitmap index because of two things in the overhead section of the entry, both of them rowids. The first rowid represents the start of a contiguous chunk of the table and the second rowid represents the end of that contiguous chunk. I point this out for two reasons: first, so you don't wonder about that; second, because it affects locking. In the overhead of a B-tree index entry there is a single lock byte. So how many index 'rows' are affected by that lock? Just the 1. In the overhead of a bitmap index entry there is a single lock byte. So how many index 'rows' are affected by that lock? All the 'rows' in that entry between the start and end rowids.
So, updating a bitmap index does not lock everything (if the 'rows' exists in mulitple data blocks), but it does lock a big chunk of rows. And since, overall, there is much less grainularity, there is much more chance of contention when trying to obtain a lock. You can use this information to extract additional knowledge about bitmap indexes. Like how even in a single user system, updates (including inserts and deletes) to a bitmap index could be very costly. Test it for yourself and your app and see. Many people will find it takes less time to drop the bitmap index, load or update your data, and then recreate the index. I know this has been a long post so instead of continuing on, I'll end it with 2 questions I think you should be able to answer based on this post: Using a bitmap index Can you index nulls? What is the impact on a bitmap indexed column if you update another, non-indexed column, but not the bitmap indexed column?
*********************** If anyone reading this example with 10 then 11 rows and wondered "Wouldn't a full table scan have been better?", I won't tell you the answer, but give yourself a gold start for thinking properly!
understanding deadlocks
Dratz Jun 19, 2007 | Comments (6) inShare
0 A client site has discovered some deadlock issues and I thought this would be a good time to talk to you about what deadlocks are, why they exist, and what you can do about them. What are deadlocks? Deadlocks occur when you have something I want and I have something you want and we are unable to share. Say we are carpenters building a house and we have two places to finish nailing up but we only have one hammer and one nail. I grab the hammer and you grab the nail. You want to nail your board and I want to nail mine. What do we do? We would probably discuss or argue the situation, each of us trying to convince the other that it's better to get one board nailed in than none. We may come to an agreement or we may not. Or we may handle it the way Oracle does: I hit you over the head with the hammer and take the nail. You see, computer processes cannot debate which board is more important to nail in. Software can only do what you tell it to do. When you give it conflicting orders (like our hammer and nail scenario), it doesn't know what the right answer is. Years ago, deadlocks would just be infinite blocks in which a DBA would have to manually kill one or both of the sessions. That meant the user process would just hang until the session(s) were killed. Now we don't have that problem. When it detects a deadlock scenario, i Oracle does something. It kills a transaction (not a session). And it kills the transaction that confirms the deadlock. Oracle rollsback the transaction and lets the other transaction continue. Some people mistakenly think that Oracle rolls back each transaction, but it doesn't (necessarily). So what should happen to the failed transaction? One of two things:
A. the app should resubmit the transaction and hope the first transaction is either done or has at least gotten it's lock (so your resubmitted transaction will be in a regular block state rather than a deadlock). or (preferably) B. the transaction is rewritten to have the same transactional path as the other transaction (i.e. you must grab the hammer before you can grab a nail). These are usually easy things to do but they aren't happening in a lot of applications for two reasons: the app doesn't handle ORA-00060 exceptions (or the called stored procedure doesn't raise them); or developers aren't given a coherent transactional roadmap, so they code things any way that makes sense to them (what does it really matter if you get the hammer or nail first, you need them both?).
Are all deadlocks code related? Well, I think ORA-00060 says they all are, but that's not the whole story, just most of it. There are (at least) a couple of other scenarios where you could get a deadlock error and then say "Wait, I'm hitting the same button as everybody else, this shouldn't be a code issue." And you may be right. Remember, a deadlock simply means I have something you want and you have something I want. Given that definition and given what we've already discussed about data blocks and bitmap indexes, it's easy to see other scenarios where deadlocks could occur. Fixing any of them is easy from a syntax perspective, but may be time consuming to implement or be detrimental to performance. Deadlocks on bitmap indexes Let's looks at this first because I think it's easier to discuss. When a client comes to me with a deadlock issue they are convinced is not code related, I ask them "Are you doing any DML on a bitmap index?" If they say "Yes", I say "Aha." You'll recall that a B-tree index has a single index entry per leaf node and that a bitmap index has hundreds or thousands of entries per leaf node (that's how they can be so powerful). So even if you only want to delete one row out of a bitmap index (or the table it's indexing), you're actually locking several bitmapped rows in the process. That's just one of the reasons why you rarely see bitmap indexes on transactional
tables. So, for bitmap indexes, it's the same basic deadlock scenario, it just might be harder to figure out if you don't understand what's really going on. Updating or deleting from bitmap indexes do not cause deadlocks, but increase the likelihood that deadlocks could occur. Deadlocks from unindexed foreign keys I don't think we've talked about constraints a lot yet (we will). A foreign key constraint basically means that you can't have a value in one table if it doesn't exist in another (parent) table. You could have an EMPLOYEE_PARKING table and put a constraint that an employee cannot be in this table unless they already exists in the EMPLOYEE table. If you declare your constraint with something like on DELETE CASCADE (if you fire the employee, you don't want them to have parking privileges anymore), then Oracle has to delete the child record in EMPLOYEE_PARKING as it deletes the record from EMPLOYEE. The problem is that if you don't have an index on that foreign key, Oracle has to keep a much broader lock on the child table to be sure it deletes all the offending rows. So, this is quite similar to the problem seen with bitmap indexes. But the fix is easy: add an index for the foreign key. Deadlocks because of low transactional space (ITL) I hope you remember during our discussion of data blocks how some space is reserved in each data block to keep track of who's interested in that block (Interested Transaction List). Some space is allocated when the block is initially created (INITTRANS). Additional space can be grabbed from the FREESPACE in the block (reread those posts if you're lost) to accomodate additional transactions up to MAXTRANS. But, if your block is full, either because of data or lot's of transactions, new transactions will have to wait for space to become available in the ITL before they can get the lock they want. When you get two such ITL waits, you can get a deadlock. The fix here is to rebuild your segment(remember tables are segments of type 'table') with a higher INITRANS and/or reserve more freespace in the blocks to handle extra transactions. I don't think this explains EVERYTHING about deadlocks, but hopefully it gives you an understanding of what they are and why they usually happen.
The Cost-Based Optimizer Part 1

0 The Oracle Cost-Based Optimizer (CBO) is an optimization engine. What that means is that it tries to find the best way to give you what you want. In this first post on the CBO, I just want to focus on the purpose of the CBO. My next post on it will explain more details. First, we must be comfortable with some basic ideas. The most important of these ideas is called equivalency. This concept may be expressed in the following example: a+b=c is equivalent to b+a=c That is a simple rewrite. Why we might want to rewrite it that way is for the next post. This post is just about things like rewrite rules. A rewrite rule means that there can be two or more ways to write logically equivalent statements. "Sally is taller than Tom" is equivalent to "Tom is shorter than Sally" as x>20 is equivalent to 20<x I hope this makes sense, because I can't think of any simpler examples. Logical equivalencies are propositions, or statements that are true for any value of the variables. We all know what variables are. Variables and rewrite rules are two of the 3 major ingredients the CBO needs to work. It is because of these two things that you will almost always hear people complain about not using "literals" and advocating using "bind variables". As Oracle has evolved, the CBO's grown a little more tolerant of literals, but hopefully you're starting to see the power of variables to the CBO (and execution plan reuse). Given all that, let's write a statement that will simulate a query for us. I don't like giving SQL examples because then you don't have to think, you just have to cut and paste. So instead of SELECT COL1, COL2..., I'm going to make our statement look like this.
"On Friday, I need to to the the grocery store, drop off some dry cleaning, mail a package, and put gas in the car." That will be our SQL statement. Now, if we gave those instructions to our Oracle robot (RobOracle), it would follow those instructions to the letter and accomplish those tasks in the order we've given them. However, that might not really be the best order in which to accomplish all of those tasks. For instance, what if we're almost out of gas? We wouldn't want to save that for the last step or we may run out of gas before we finish the other tasks. But, if we had some gas in the car and were preparing for a long roadtrip, it may make sense to leave it as the last task so we have a full tank for our trip. And where are all these places? Will we be zigzagging across town instead of taking one simple path? Are we going to be buying ice cream at the store? Will it melt if we buy it first and leave it in our hot car? These are issues that we intuitively resolve for ourselves (unless we're 17 and just got our license). But RobOracle can't do that. Why not? It doesn't have enough information. Without enough information, it can't optimize its tasks. Back in the world of SQL, we have a very easy way to give Oracle that information: statistics. Statistics tell Oracle things about the objects you are querying, like number of distinct values in a column, avg number of rows per data block, avg row size, number of leaf blocks in an index, cpu utilization... All kinds of things that gives the CBO enough information to estimate the best way to achieve your results. We'll talk about how to get and maintain those stats in the next post. You tell Oracle what you want, and Oracle will apply rewrite rules to come up with the best logical equivalency it can so you can get your results at the cheapest cost without changing your results. The CBO keeps you from zigzagging all over your database.
Recursive Calls
Dratz Nov 6, 2007 | Comments (10) inShare
0 Before I get to the next part on the CBO, I wanted to address a reader's question about recursive calls. When tuning SQL with TKPROF or a similar tool, you will often find statistics for "recursive" calls. WHAT ARE RECURSIVE CALLS? In Oracle, a recursive call is a call, for example, that must be completed before the user's SQL can be completed. Say you wanted to order a pizza for delivery. You make one call, place your order, and wait for your pizza. You do NOT call the kitchen staff directly and tell them how to make your pizza , call the cashier to give him your credit card number, call the delivery man and tell him to pick up your pizza and bring it to your house.
Nope, you make one call, not three. The person you place your order with makes those "recursive" calls to the kitchen and delivery staff to make sure your order is complete. The kitchen staff may make additional "recursive" calls ("Where are the mushrooms?") as may the delivery man ("Where is that street located?"). You don't care about these details, you just want your pizza. WHAT KINDS OF RECURSIVE CALLS DOES ORACLE MAKE? As you can tell from our pizza example, Oracle will make a recursive call whenever it has to. Some examples of when it "has to" include: you want to insert data into a DMT, but all extents are full and news ones need to be allocated to hold your data you're doing something on a table that causes a trigger to fire (e.g. - don't allow inserts if it's Saturday)
performing DDL data dictionary cache needs info about objects during parsing (yikes!) you have PL/SQL units that include SQL statements (like a stored procedure that inserts into a table, or a function that selects from a table) TUNING RECURSIVE CALLS??? I hope you can see from this brief discussion that the concept of tuning recursive calls is not as easy as configuring the Recursive Program Interface (RPI). Tuning depends on several factors, the two biggest being what are you trying to do and what version are you on? Sometimes recursive calls indicate a problem, sometimes they don't. Can you see why? I hear and am asked about the notion that recursive calls should always be less than "real" calls. I ask "Why?" If I have a db app that only accesses the db via stored procedures and each of my procedures issues an average of 10 DML statements within it, then why would a 10/1 ratio be bad? But that's not to say that analysing recursive calls is not an important tuning step. You need to know what your database is doing and why. Recursive calls could be a hidden problem. Just don't freak out when you order your pizza from a lady, but a guy delivers it.

How Oracle Works

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

How Oracle Works

Uploaded by

Copyright:

Available Formats

Why Oracle Works the Way it Does Introduction

Dratz Apr 25, 2006 | Comments (6) inShare

Segments and Extents

Data Blocks Part 1

Data Blocks Part 2

Basic Memory Structures

Intro to Oracle Processes

DEDICATED VS. SHARED

SHOULD YOU USE DEDICATED OR SHARED SERVERS?

A note about RMAN

What happens when you run a SQL statement?

The basic SQL statement

Excessive Redo Generation

The Cost-Based Optimizer Part 1

You might also like