[aprssig] High speed datbase inserts (CGI)

[AE5PL Lists]

I agree, Ken.  Even on an "underpowered" system (I am populating a MS
SQL database on a mirrored drive configuration with a 2.4 GHz CPU which
is, IMO, underpowered for this application), insertion directly into the
database is probably the best route.

A couple of caveats:

Use minimal indexes.  Indexes are more performance problematic than most
other factors for databases which are mostly added to instead of
searched.

Bulk backups for large databases can lock out the updating program for
extended periods of time.  This could be an issue, depending on buffer
depths in the socket connection and the database driver.  Also
timestamps can be off unless you timestamp the data upon receipt instead
of at time of row insertion.

We have people using javAPRSSrvr's database interface to populate mySQL,
PostgreSQL, and MS SQL Server databases from the APRS-IS feed with
relatively good performance.  This interface uses the Java database
drivers (JDBC) and appears to work fairly well under load.  It is my
understanding that there are similar drivers for Perl.

73,

Pete Loveall AE5PL
mailto:pete at ae5pl.net 

> -----Original Message-----
> From: Ken Linder
> Posted At: Tuesday, March 01, 2005 1:21 AM
> Subject: Re: [aprssig] High speed datbase inserts (CGI)
> 
> I might not be a findu expert but I manage my share of data 
> (> 3 TB) at work.  I would say that it would probably be 
> easier to just shove the data into mySQL.
> 
> UNLESS...  Your mySQL server is quite underpowered.  Then, it 
> might be better to use the temporary files and let the mySQL 
> engine 'breate'.