Page 1 of 1 1
Topic Options
#193552 - 2009-04-22 11:30 PM Any way of speeding up an import from Excel to Access...
Viggen Offline
Starting to like KiXtart

Registered: 2002-03-22
Posts: 110
Hi all!

Got a little speed issue here... \:\)


When I run the following script it takes 14.5 minutes to import 18200 lines from a csv file to an Access DB.

If I do the Import from Access it takes less than 5 sec. to do all 18200 lines \:\(


 Code:
break on

$nul = setoption('WrapAtEOL','On')

?? "[1] : " + @time

$K2E_EXEPath = @scriptdir
$lokaldb = $K2E_EXEPath + "\Artiklar.mdb"
$importfile = loadfile(@scriptdir + '\090420.csv')
$i_array = quicksort($importfile,0)

?? "[2] : " + @time

$ConnDSN = "Driver={Microsoft Access Driver (*.mdb)};Dbq=" + $lokaldb + ";Uid=;Pwd=;"
$sql = "CREATE TABLE tmptbl (Artikelnummer varchar (255), Benämning varchar (255), Styckpris double, Flag char (1), EAN varchar (255))"
$rs = DBCommand($ConnDSN,$sql)

?? "[3] : " + @time

For each $row in $i_array
  If Not $row = ''
    $items = Split($row,";")

    If not Instr($items[0],'Artikel')
      $aitem = Cstr($items[0])

      $bitem = Cstr($items[1])
      If Instr($bitem,"'")
        $bitem = Join(Split($bitem,"'"),"")
      Endif

      $citem = Join(Split($items[2],' '),'')

      $ditem = Cstr($items[3])

      $values = "'" + $aitem + "','" + $bitem + "','" + $citem + "','" + $ditem + "'"
      $fields = "Artikelnummer,Benämning,Styckpris,EAN"
      $sql = "INSERT INTO tmptbl(" + $fields + ") VALUES(" + $values + ")"
      $rs = DBCommand($ConnDSN,$sql)

      $items = ''

    Endif
  Endif
Next

?? "[4] : " + @time

Get $x




I have looked at the example here, that looks like this:
(would be nice to get the info from the xls, instead of saving it to a csv first)

 Code:
 Dim cn As ADODB.Connection
    Dim strSQL As String
    Dim lngRecsAff As Long
    Set cn = New ADODB.Connection
    cn.Open "Provider=Microsoft.Jet.OLEDB.4.0;" & _
        "Data Source=C:\test\xltestt.xls;" & _
        "Extended Properties=Excel 8.0"
    
    'Import by using Jet Provider.
    strSQL = "SELECT * INTO [odbc;Driver={SQL Server};" & _
        "Server=<server>;Database=<database>;" & _
        "UID=<user>;PWD=<password>].XLImport9 " & _
        "FROM [Customers$]"
    Debug.Print strSQL
    cn.Execute strSQL, lngRecsAff, adExecuteNoRecords
    Debug.Print "Records affected: " & lngRecsAff
        
    cn.Close
    Set cn = Nothing


But I can't figure out how to get the strSQL-line to work with DBCommand()

Or is there another way to get closer to the 5 sec. result that I get in Access?

/Viggen

Top
#193554 - 2009-04-23 01:43 AM Re: Any way of speeding up an import from Excel to Access... [Re: Viggen]
Howard Bullock Offline
KiX Supporter
*****

Registered: 2000-09-15
Posts: 5809
Loc: Harrisburg, PA USA
Which section of your code is slow? You have 4 outputs for time, please post the output results so we can see how long each section takes.
_________________________
Home page: http://www.kixhelp.com/hb/

Top
#193555 - 2009-04-23 01:54 AM Re: Any way of speeding up an import from Excel to Access... [Re: Howard Bullock]
Howard Bullock Offline
KiX Supporter
*****

Registered: 2000-09-15
Posts: 5809
Loc: Harrisburg, PA USA
 Quote:
$importfile = loadfile(@scriptdir + '\090420.csv')
$i_array = quicksort($importfile,0)


The loadfile UDF I found can load a file into an array ir directly into a variable. You did not post your UDF code so I can not determine if you intended read the file into a variable or into an array. However, the quicksort UDF is used as if you expected that $importfile was an array.

 Code:
Function loadfile($file, optional $array, $Uni)
	DIM $fso,$f,$fs
	if $uni $uni = $uni-3 else $uni = -2 endif

	if not $uni $uni = -2 endif
	$fso = CreateObject("Scripting.FileSystemObject")
	$f = $fso.GetFile($file)
	If @ERROR Exit 2 EndIf
	$fs = $f.OpenAsTextStream(1,$uni)
	if not $array
		$loadfile = $fs.Read($f.size)
	else
		$loadfile = Split($fs.Read($f.size),$array)
	endif
	Exit @ERROR 
EndFunction
 

 Code:
function quicksort($a, optional $indexcol )
    ;$a                        array to be sorted 
    ;$indexcol                 index of subitem to sort on 
 
_________________________
Home page: http://www.kixhelp.com/hb/

Top
#193556 - 2009-04-23 03:04 AM Re: Any way of speeding up an import from Excel to Access... [Re: Howard Bullock]
Viggen Offline
Starting to like KiXtart

Registered: 2002-03-22
Posts: 110
Ah! my bad. sorry

It is the "For each $row... Next"-part that is slow.

The Loadfile that I use looks like this:
 Code:
Function loadfile($file)
  DIM $fso,$f,$fs

  $fso = CreateObject("Scripting.FileSystemObject")
	$f = $fso.GetFile($file)
	If @ERROR
    Exit(2)
  EndIf

  $fs = $f.OpenAsTextStream(1)
  $loadfile = Split($fs.Read($f.size),@CRLF)

  Exit(@ERROR)
EndFunction


The output looks something like this:

[1] : 02:31:04
[2] : 02:31:11
[3] : 02:31:11
[4] : 02:43:08

[3] to [4] takes 12 mins.
(I have removed some data from the csv)

The thing is that in the same time as this script go from [1] to [3], Access has imported everything.

That's why I wonder if the "SELECT * INTO"-approach would be faster.


/Viggen

Top
#193560 - 2009-04-23 12:13 PM Re: Any way of speeding up an import from Excel to Access... [Re: Viggen]
BradV Offline
Seasoned Scripter
****

Registered: 2006-08-16
Posts: 687
Loc: Maryland, USA
Can you build up all the insert commands into a single one and then just make one call to the database to do the insert? In my experience, making the COM call invokes a time penalty. It works, but isn't very fast. If you can build up your command a make a single call, or maybe 100 calls instead of eighteen thousand?

Regards,

Brad

Top
#193561 - 2009-04-23 12:48 PM Re: Any way of speeding up an import from Excel to Access... [Re: BradV]
Viggen Offline
Starting to like KiXtart

Registered: 2002-03-22
Posts: 110
That sounds like a plan, but...

I have no clue on how to do that \:\(

/Viggen

Top
#193562 - 2009-04-23 01:43 PM Re: Any way of speeding up an import from Excel to Access... [Re: Viggen]
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4402
Loc: New Jersey
I'm not sure why you'd be scripting something that can be done directly, unless you're using a script to modify the data in some way. If you just modify the data, how long does that take? Write the modified data to a file and trigger an import command rather than feeding each record individually.

I collect event log data from around 80 crucial servers each night.. the script I wrote filters the raw data and converts it to a format I need, stripping out certain records based on content or record type. The resulting files are then fed to a SQL server for import. Hundreds of thousands of lines of data being processed and it runs in about 90 minutes. When I tried writing directly to the SQL server, it took over 7 hours.

Glenn
_________________________
Actually I am a Rocket Scientist! \:D

Top
#193563 - 2009-04-23 01:49 PM Re: Any way of speeding up an import from Excel to Access... [Re: Viggen]
Howard Bullock Offline
KiX Supporter
*****

Registered: 2000-09-15
Posts: 5809
Loc: Harrisburg, PA USA
Using ODBC to insert a record at a time will be painful. The approach you nned to take is to leverage the programability of Access itself. You should be able to find ways to utilize Access' import wizard functionality via script. The result should be import speeds the same as interactive Access use.

I did not find any quick access to the importWizard although I did see some references.

Using ADO may be faster: How Can I Import a .CSV File into an Access Database?
_________________________
Home page: http://www.kixhelp.com/hb/

Top
#193582 - 2009-04-24 07:44 AM Re: Any way of speeding up an import from Excel to Access... [Re: Howard Bullock]
Viggen Offline
Starting to like KiXtart

Registered: 2002-03-22
Posts: 110
The reason for scripting this is that, the client-computer has no MS Office.

I followed that link Howard, and I can only say, OH... My... God!

I have changed the script and now the time output looks like this \:\)

[1] : 07:35:25
[2] : 07:35:33
[3] : 07:35:33
[4] : 07:35:36

From 12 Mins. to 11 Sec.

Now that is a very good way to "speeding up an import..."

Thank you very much!

/Viggen

Top
Page 1 of 1 1


Moderator:  Glenn Barnas, NTDOC, Arend_, Jochen, Radimus, Allen, ShaneEP, Ruud van Velsen, Mart 
Hop to:
Shout Box

Who's Online
0 registered and 476 anonymous users online.
Newest Members
batdk82, StuTheCoder, M_Moore, BeeEm, min_seow
17885 Registered Users

Generated in 0.062 seconds in which 0.028 seconds were spent on a total of 13 queries. Zlib compression enabled.

Search the board with:
superb Board Search
or try with google:
Google
Web kixtart.org