14ecmwfbufr Software

BUFR software
Milan Dragosavac
Slide 1
WMO BUFR training course
Langen
17-20 April 2007
Slide 1
ECMWF Data Formats

Standard WMO data formats ASCI and binary
FM 94-XII Ext. BUFR
FM 92-XII Ext. GRIB

BUFR for observations and verification scores
GRIB for fields
Archiving formats BUFR and GRIB only
Slide 2
Langen
17-20 April 2007
Slide 2
Requirements for BUFR software

Suitable for all observations data handling
Building blocks for other applications
Encoding
Decoding
Low maintenance costs
Portable
User friendly
Slide 3
Langen
17-20 April 2007
Slide 3
Programming languages used

Fortran 77
- All BUFR subroutines and functions
C
- Reading/writing BUFR data
- gbyte, sbyte routines to do bit manipulations
Slide 4
Langen
17-20 April 2007
Slide 4
Ecmwf BUFR software

How to get the software?
The software can be downloaded from Ecmwf web site
http://www.ecmwf.int/products/data/software/
Go to Free download and get bufr_000320.tar.gz file
gunzip bufr_000320.tar.gz
cd bufr_000320
Read README file for explanations
Execute make command which will create libbufr.a
cd examples
Slide 5
Running make will create number of executables

Langen
17-20 April 2007
Slide 5
Ecmwf BUFR software

Content of bufr_000320 directory
-rwxr--r--
1 maa ma
462 Jul 19 2006 Makefile
-rwxr--r--
1 maa ma
467 Jul 19 2006 Makefile.in
-rwxr--r--
1 maa ma 11201 Jul 19 2006 README
drwxr-xr-x 2 maa ma 4096 Jan 22 17:13 bufr_main_c_program

drwxr-xr-x 2 maa ma 4096 Feb 14 08:35 bufrdc
drwxr-xr-x 2 maa ma 8192 Feb 14 08:35 bufrtables
-rwxr--r--
1 maa ma 5327 Jul 19 2006 build_library
drwxr-xr-x 2 maa ma 4096 Feb 7 07:52 config

Slide 6
Langen
17-20 April 2007
Slide 6
Ecmwf BUFR software

drwxr-xr-x 2 maa ma 4096 Jan 5 11:42 data
drwxr-xr-x 2 maa ma 4096 Jul 19 2006 doc
drwxr-xr-x 2 maa ma 4096 Feb 14 08:28 examples
-rwxr--r--
1 maa ma
1330 Jul 19 2006 install
-rwxr--r--
1 maa ma
1196 Jul 19 2006 licence
drwxr-xr-x 2 maa ma 4096 Jan 22 16:35 options

drwxr-xr-x 2 maa ma 4096 Feb 14 08:35 pbio
Slide 7
Langen
17-20 April 2007
Slide 7
Ecmwf BUFR software

config directory contains various configuration files
for different platforms and compilers
options directory contains various options for different
platforms
You can make additional configurations and options if

needed
Slide 8
Langen
17-20 April 2007
Slide 8
Ecmwf BUFR software

pbio
pbio directory contain all c and fortran routines needed
for BUFR and CREX IO
all routines are part of bufr and crex libraries
bufrdc
bufrdc directory contain all bufr software source code
routines which are written in fortran 77
Slide 9
Langen
17-20 April 2007
Slide 9
Ecmwf BUFR software

examples
decode_bufr.F
tdexp.F
bufr2crex.F
create_bufr.F
doc
doc directory contain bufr software Users Guide and
Reference Manual in pdf format
Slide 10
Langen
17-20 April 2007
Slide 10
Bufr format
Binary universal form for the representation of
meteorological data
Indicator section
Identification section
Optional section
Data description section

Data section
End section
Slide 11
Langen
17-20 April 2007
Slide 11
Bufr indicator section

Octet number Content
1-4
BUFR four letters in CCITT IA5
5-7
Total length of Bufr message in bytes
Bufr edition number ( currently 4 )
Slide 12
Langen
17-20 April 2007
Slide 12
Identification section
1-3
Length of section 1
Bufr master table (zero if WMO FM 94-XII used )
5-6
Originating / generating centre
7-8
Originating / generating sub-centre

Slide 13
Langen
17-20 April 2007
Slide 13
Identification section continued

9
Update sequence number (zero for original BUFR

message; incremented by one for updates)
10
Bit 1 = 0 no optional section

Bit 1 = 1 optional section follows
Bit 2-8 set to zero ( reserved )
11
Data category ( Bufr table A )
12
International data sub-category (common Tab 13)
13
Local sub-category
Slide 14
Langen
17-20 April 2007
Slide 14
Identification section continued

Octet number
Content
14
Version number of master table used
15
Version number of local tables used ( set to 0 or

255 if no local tables used
16 - 17
Year ( 4 digits )
18
Month
19
Day
20
Hour
21
Minute
22
Second
23-
Slide 15
Reserved for local use by ADP centres
Langen
17-20 April 2007
Slide 15
Optional section
1-3
Length of section in bytes
Set to zero ( reserved )
5-
Reserved for local used by ADP centres
Slide 16
Langen
17-20 April 2007
Slide 16
Data description section

1-3
4
5-6
7

Number of data subsets
Bit 1 = 1 observed data
Bit 1 = 0 other data
Bit 2 = 1 compressed data
Bit 2 = 0 non compressed data

Bits 3-8 set to zero ( reserved )
17
Collection of descriptors Slide
defining
the data
Langen
17-20 April 2007
Slide 17
Data section
1-3
5-
Binary data as defined by sequence descriptors
Slide 18
Langen
17-20 April 2007
Slide 18
End section
1-4
7777 ( coded according to CCITT IA5 )
Slide 19
Langen
17-20 April 2007
Slide 19
Ecmwf Bufr software

Bufr is table driven code form
- no need to transfer element names, units
- no need to change bufr software for new observations

Bufr table B classification elements
- reference number
- element name
- element unit
- scale
- reference value
- data width in bits
Langen
Slide 20
17-20 April 2007
Slide 20
Ecmwf Bufr software

Bufr table D list of common sequences to describe the
data
- table B elements
- operators
- other table D descriptors
Bufr table C text and meaning of code/flag tables
Slide 21
Langen
17-20 April 2007
Slide 21
Ecmwf Bufr software

Bufr table naming convention
Bssswwwwwxxxxxyyyzzz.TXT
Cssswwwwwxxxxxyyyzzz.TXT
Dssswwwwwxxxxxyyyzzz.TXT
sss
- Master table number ( zero for WMO

meteorological tables
wwwww - Originating sub-centre

xxxxx
- Originating centre
yyy
- Version number of master tables used

Slide 22
zzz
- Version number of local tables used
Langen
17-20 April 2007
Slide 22
Ecmwf Bufr software

PATH for Bufr tables
export BUFR_TABLES=/xxx/
setenv BUFR_TABLES /xxx/
The path must end with back slash /
Defaults
Integer missing value indicator
NVIND=2147483647
Real missing value indicator
Slide 23
RVIND= 1.7E38
Langen
17-20 April 2007
Slide 23
Decoding Bufr data

There are four steps in bufr data processing
Open bufr input file

Read in one bufr message at a time
Expand bufr message
Use the data
Repeat last three steps until the end of file is reached
Slide 24
Langen
17-20 April 2007
Slide 24
PBIO routines
FORTRAN routines to handle products in an unblocked binary file
subroutine PBOPEN ( kunit, filename, mode, kerr )

Input arguments:
filename - character variable
mode
- character variable r read, w write, a append, r+ read and

write
Output arguments:
kunit - integer unit number
kerr - integer status return code

Slide 25
Langen
17-20 April 2007
Slide 25
PBIO routines
kerr - 0, successful
-1, file does not exist
-2, invalid file name

-3, invalid open mode specified
Slide 26
Langen
17-20 April 2007
Slide 26
PBIO routines
To read Bufr product
subroutine PBBUFR (kunit, karray, kinlen, koutlen, kret)

Input parameters are integers:
kunit
- unit number obtained from pbopen
karray - array big enough to hold the product

kinlen - size in bytes of the array
Output parameters are integers:

koutlen - actual size in bytes of the product read into karray
kret
- status return code
Langen
17-20 April 2007
Slide 27
Slide 27
PBIO routines
kret from pbbufr subroutine:
Product has been successfully read
-1
End-of-file is hit before a product is read.

(Note that EOF does not cause a program fail.
This value must be explicitly caught by the
caller to avoid looping at EOF)
-2
There has been an error in the file-handling

(e.g. the file contains a truncated product)
-3
The size of karray is not sufficient for the

Slide 28
product
Langen
17-20 April 2007
Slide 28
PBIO routines
subroutine PBCLOSE (kunit, kret)
Input argument is an integer:

kunit - unit number from PBOPEN
Output argument is an integer:
kret status return code
0 - OK
-1 - error in handling the file
Slide 29
Langen
17-20 April 2007
Slide 29
PBIO routines
File handling FORTRAN subroutines
subroutine PBSEEK (kunit, koffset, kstart, kret)

Input arguments are integers:
kstart - 0, if koffset is an absolute count from the
beginning of the file
1, if koffset is a relative offset from the current
byte position in the file
2, if koffset is an absolute offset from the end of

Slide 30
file
Langen
17-20 April 2007
Slide 30
PBIO routines
Output argument is an integer:
kret - status return code
>= 0 Byte offset from the start of file after positioning

-1 End-of-file is hit before the product is read
-2 Error in handling the file
Slide 31
Langen
17-20 April 2007
Slide 31
PBIO routines
subroutine PBREAD (kunit, karray, kount, kret)

kount - number of bytes to read from the file
Output arguments:
karray - an integer array to accept the bytes from the
read
>=0 number of bytes read

Slide 32
-1
EOF reached before the product is read
-2
Error in file handling
Langen
17-20 April 2007
Slide 32
PBIO routines
subroutine PBWRITE( kunit, karray, kount, kerr)

kunit unit number from PBOPEN
karray- an integer array holding bytes for write
kount number of bytes to write to the file
Output arguments:
>= 0 number of bytes written to theSlide

file
33
-1
Error in writing to the file
Langen
17-20 April 2007
Slide 33
Bufr software FORTRAN routines

Bufr expansion
subroutine BUFREX( kbufl, kbuff, ksup, ksec1, ksec2,

ksec3, ksec4, kelem, cnames,
cunits, kvals, values, cvals, kerr)
Input arguments:
kbufl
- length of bufr message in words
kbuff
- an integer array containing bufr message
kelem - expected number of expanded elements
kvals - the size of values array in Slide

words
34
Langen
17-20 April 2007
Slide 34

Output arguments:
ksup
- an array containing supplementary information
ksec0
- an array containing section 0 information
ksec1
ksec2
ksec3
ksec4
cnames - character array containing element names

cunits
- character array containing element units
values - real*8 array containing element

values
Slide 35
kerr
- status return code
Langen
17-20 April 2007
Slide 35

ksec0: an integer array ( size of 3 )
Array
index
Content
Length of section 0 in bytes
Total length of bufr message in bytes
Slide 36
Langen
17-20 April 2007
Slide 36

ksec1: an integer array ( size of 40 )
Array
index
Content
Origination centre
Update sequence number
Flag ( presence of section 2 in the message )
Bufr message type ( Bufr table A )
Slide 37
Langen
17-20 April 2007
Slide 37

ksec1 continued
Array
index
Content
Bufr message sub-type ( local use)
Version number of local tables used
Year
10
Month
11
Day
12
Hour
13
Minute
Slide 38
Langen
17-20 April 2007
Slide 38

ksec1 continued
Array
index
Content
14
Bufr master tables used
15
Version number of master table used
16
Originating sub-centre
17
International sub-category
18
Second
19-
Local ADP centre information (byte by byte)

Slide 39
Langen
17-20 April 2007
Slide 39

ksec2
Array
index
Content
2-
RDB key in packed form
Slide 40
Langen
17-20 April 2007
Slide 40

ksec3
Array
index
Content
Reserved
Number of subsets
Flag ( data type, compression)
Slide 41
Langen
17-20 April 2007
Slide 41

ksec4
Array
index
Content
2-
Reserved
Slide 42
Langen
17-20 April 2007
Slide 42

ksup an integer array (size 9) containing supplementary
information
Array index
Content
Dimension ksec1 array
Total number of expanded elements
Number of subsets
Total number of elements in cvals array
43
Total bufr message length in Slide
bytes
Dimension of ksec0 array
Langen
17-20 April 2007
Slide 43

Values array is of type real*8
If bufr table B element is CCITTIA5, corresponding
element contain a real number, when truncated to an
integer represents
index*1000+length
where
index subscript of the element in CVALS array where

character strings are stored
length number of characters in the string
Slide 44
Langen
17-20 April 2007
Slide 44
In the case of multi-subset data, one dimensional array
VALUES contains all subsets. Index to values array of

i-th element of observation is:
index=i+(nsub-1)*kelem
Slide 45
Langen
17-20 April 2007
Slide 45
BUFR SOFTWARE fortran ROUTINES

Bufr encoding
subroutine BUFREN (ksec0, ksec1, ksec2, ksec3, ksec4,

ktdlen, ktdlst, kdlen, kdata, kelem,
kvals, values, cvals, kbufl, kerr)
Input arguments:
ksec0
- an integer array containing section 0 information
ksec1
- an integer array containing section 1 information
ksec2
- an integer array of 4096 words containing section

2 information
Slide 46
ksec3
- an integer array containing section 3 header

information
Langen
17-20 April 2007
Slide 46

ksec4
- an integer array containing section4 header

information
ktdlen - an integer, number of data descriptors to be

packed in section 3 of bufr message
ktdlst
- an integer array containing ktdlen descriptors
kdlen
- an integer, dimension of kdata array
kdata
- an integer array containing delayed replication

values and possibly new reference value if
required
kelem - an integer containing expected number of

expanded elements
kvals
Slide 47
- an integer containing expected number of data

values
Langen
17-20 April 2007
Slide 47

values - real*8 array of kvals words containing element
values
cvals
- character*80 array of kvals words containing

CCITTIA5 elements
Output arguments:
kbufl - an integer containing the length of bufr message

in words
kbuff - an integer array containing packed bufr message
kerr
- integer, status return code

Slide 48
Langen
17-20 April 2007
Slide 48

Printing routines
Subroutine BUPRS0 (ksec0)

Subroutine BUPRS1 (ksec1)
Section 2 is for local use no general printing routines
Subroutine buukey (ksec1,ksec2,key,ksup,kerr)
Subroutine BUPRS2 (ksup, key)
Slide 49
Langen
17-20 April 2007
Slide 49

To print section3
Subroutine BUSEL (ktdlen, ktdlst, ktdexl, ktdexp, kerr)

Or
Subroutine BUSEl2(ksubset, kelem, ktdlen, ktdlst, ktdexl,
ktdexp, cnames, cunits, kerr)
Subroutine BUPRS3 (ksec3, ktdlen, ktdlst, ktdexl, ktdexp,
kelem, cnames)
Slide 50
Langen
17-20 April 2007
Slide 50

To print data
Subroutine BUPRT (k, ksub1,ksub2,kelem,cnames,cunits,

cvals, kvals, values, ksup, ksec1, kerr)
Useful subroutines
Subroutine BUS012 (kbufl, kbuff, ksup, ksec0, ksec1,
ksec2, kerr)
Subroutine BUS0123 (kbufl, kbuff, ksup, ksec0, ksec1,

Slide 51
ksec2, ksec3, kerr)
Langen
17-20 April 2007
Slide 51

Bufr template design
Basic idea is to know one to one correspondence
between data descriptors expanded and the data values
To achieve that use:
subroutine BUXDES ( k, ksec1, ktdlen, ktdlst, kdlen,

kdata, kelem, ktdexl, ktdexp,
cnames, cunits, kerr)
Input arguments:
k
- an integer, 0 no print; 1 print Slide 52
Langen
17-20 April 2007
Slide 52

ksec1 - an integer array containing section 1 information.
Set the following words
ksec1(2) Bufr edition number

ksec1(3) Originating centre
ksec1(8) Version number of local tables used
ksec1(15) version number of master table used
ktdlen number of data descriptors
ktdlst- list of data descriptors to be packed in section 3
kdlen dimension of kdata array
kdata - an integer array containing delayed

replications
Slide 53
kelem expected number of expanded elements
Langen
17-20 April 2007
Slide 53
|Bufr software FORTRAN routines

Output arguments:
ktdexl number of expanded elements

ktdexp integer array containing list of expanded
elements
cnames character *64 array containing element names
cunits character*24 array containing element units
kerr status return code
Slide 54
Langen
17-20 April 2007
Slide 54

Program to use for template design
/bufr_000320/examples/tdexp.f
or
Bufrviewer application
Slide 55
Langen
17-20 April 2007
Slide 55

Resolving bit maps
- Bit maps are very useful bufr features which associate

quality control, first order statistics, departures,
retained/replaced and substituted values with original
values
- One application might have many bitmaps referring
backward to the data
Slide 56
Langen
17-20 April 2007
Slide 56
subroutine BUBOX (ksub, ksup, kelem, kwtr, cnames,

cunits, kvals, values, kbox, kapp, klen,
kboxr, vals, cboxn, cboxu, kerr)
ksub subset number
ksup array containing supplementary information
kelem expected number of elements, it must be the same

as used in call to bufrex
kwtr an integer array containing list of bufr table B
reference numbers ( ktdexp, output from busel
routine)
Slide 57
Langen
17-20 April 2007
Slide 57

cnames array containing elements names
cunits - array containing element units
kvals expected number of data values

values array containing expanded values
Output arguments:
kbox number of elements in the first column of box
kapp - number of applications
klen - max index for number of rows. The next column
starts at KLEN +1 element or
Slide 58
index=i +(KAPP-1)*klen
Langen
17-20 April 2007
Slide 58

kboxr array containing bufr table B reference numbers
vals a real*8 array containing boxed values
cboxn character*64 array, boxed element names

cboxu character*24 array, boxed element units
kerr status return code
Slide 59
Langen
17-20 April 2007
Slide 59

subroutine buprtbox(kbox, kapp, klen, kboxr, vals,
cboxn, cboxu)
Input arguments:
kbox
- an integer containing number of elements in

the first column of the box
kapp
- an integer containing number of applications
klen
- an integer containing max index for number of

rows. The next column starts at klen+1
element or index =i+(kapp-1)*klen

Slide 60
Langen
17-20 April 2007
Slide 60

kboxr
- an integer array containing bufr table B

reference numbers
vals
- a real*8 array containing boxed values
cboxn
- a character*64 array containing boxed names
cboxu
- a character*24 array containing boxed units
Slide 61
Langen
17-20 April 2007
Slide 61

Example output
1 OPERATOR
************** 222000.0
2 GENERATING CENTRE( CODE TABLE 00
**************
98.0
3 GENERATING APPLICATION (CODE TAB
**************
1.0
4 STATISTICS (008024/008023)
**************
********
5 INCREMENTAL UPDATE NUMBER
**************
********
6 MINIMISATION SIMULATION NUMBER
**************
********
10.0
70.0
320.0
70.0
7 WMO BLOCK NUMBER
8 WMO STATION NUMBER
Langen
Slide 62
17-20 April 2007
Slide 62
Some Bufr operators

Quality control information in Bufr
- class 33 element
- associated field
- Operator 222000 + bit map
- 033220 variation analysis report event , valid for whole
report( observation)
Slide 63
Langen
17-20 April 2007
Slide 63
Some Bufr operators

Associated field
204001
031021
011001
204000
After expansion
031021 associate field significance
999999 associated field
Slide 64
011001 wind direction

Langen
17-20 April 2007
Slide 64
Some Bufr operators

Quality control information using 222000 operator
222000
101005
031031
001031
001032
101005
033007
Slide 65
Langen
17-20 April 2007
Slide 65
Some Bufr operators

First order statistics
224000
237000 Use previously define bit map
001031
001032
008023 First order statistics
101005
224255 First order statistics marker
Slide 66
Langen
17-20 April 2007
Slide 66
Some Bufr operators

Difference statistical values
225000
237000 Use previously define bit map
001031
001032
008024 Difference statistics
101005
225255 Difference statistics marker
Slide 67
Langen
17-20 April 2007
Slide 67
Bufr software and pre-processing

What happens when bufr data reach DPC
How to organize observation
Pre-process observations
- unpacking
- reorganization of data
- possible quality control
- grouping of observations which belong to
particular analysis cycle
- creating RDB key for further more efficient processing
- packing the data into bufr
Slide 68
- writing data into RDB

Langen
17-20 April 2007
Slide 68
Bufr software and pre-processing

Alternatively
- unpack bufr message
- write unpacked values into DB

- make decisions about archiving
Slide 69
Langen
17-20 April 2007
Slide 69

14ecmwfbufr Software

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

14ecmwfbufr Software

Uploaded by

Copyright:

Available Formats

BUFR software

WMO BUFR training course

17-20 April 2007

ECMWF Data Formats

FM 92-XII Ext. GRIB

WMO BUFR training course

17-20 April 2007

Requirements for BUFR software

WMO BUFR training course

17-20 April 2007

Programming languages used

WMO BUFR training course

17-20 April 2007

Ecmwf BUFR software

Running make will create number of executables

17-20 April 2007

Ecmwf BUFR software

462 Jul 19 2006 Makefile

467 Jul 19 2006 Makefile.in

1 maa ma 11201 Jul 19 2006 README

drwxr-xr-x 2 maa ma 4096 Jan 22 17:13 bufr_main_c_program

1 maa ma 5327 Jul 19 2006 build_library

drwxr-xr-x 2 maa ma 4096 Feb 7 07:52 config

WMO BUFR training course

17-20 April 2007

Ecmwf BUFR software

1330 Jul 19 2006 install

1196 Jul 19 2006 licence

drwxr-xr-x 2 maa ma 4096 Jan 22 16:35 options

WMO BUFR training course

17-20 April 2007

Ecmwf BUFR software

You can make additional configurations and options if

WMO BUFR training course

17-20 April 2007

Ecmwf BUFR software

WMO BUFR training course

17-20 April 2007

Ecmwf BUFR software

WMO BUFR training course

17-20 April 2007

Data description section

WMO BUFR training course

17-20 April 2007

Bufr indicator section

BUFR four letters in CCITT IA5

Total length of Bufr message in bytes

Bufr edition number ( currently 4 )

WMO BUFR training course

17-20 April 2007

Bufr master table (zero if WMO FM 94-XII used )

Originating / generating centre

Originating / generating sub-centre

WMO BUFR training course

17-20 April 2007

Identification section continued

Update sequence number (zero for original BUFR

Bit 1 = 0 no optional section

Data category ( Bufr table A )

International data sub-category (common Tab 13)

WMO BUFR training course

17-20 April 2007

Identification section continued

Version number of master table used