SitePoint Sponsor

User Tag List

Results 1 to 4 of 4
  1. #1
    SitePoint Addict
    Join Date
    Mar 2005
    Posts
    231
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Data Import Tools - Large Database

    So I am importing a csv file that is about 2GB large with approximately 8 million regards using Toad and got the following error:

    Code:
    System.OutOfMemoryException
    Exception of type 'System.OutOfMemoryException' was thrown.
    Stack Trace:
       at System.Text.RegularExpressions.RegexRunner.DoubleTrack()
       at System.Text.RegularExpressions.RegexRunner.EnsureStorage()
       at System.Text.RegularExpressions.RegexInterpreter.Goto(Int32 newpos)
       at System.Text.RegularExpressions.RegexInterpreter.Go()
       at System.Text.RegularExpressions.RegexRunner.Scan(Regex regex, String text, Int32 textbeg, Int32 textend, Int32 textstart, Int32 prevlen, Boolean quick)
       at System.Text.RegularExpressions.Regex.Run(Boolean quick, Int32 prevlen, String input, Int32 beginning, Int32 length, Int32 startat)
       at System.Text.RegularExpressions.Match.NextMatch()
       at Quest.Toad.Db.SqlFunction.EvaluateSql(Connection connection, String sql, Object[] parameters, IDataParameter[]& outparameters, Int32& inParameterIndex, Int32& outParameterIndex, Boolean function)
    What do you think caused this?
    What other alternatives are out there for a database of this magnitude?

  2. #2
    SitePoint Addict
    Join Date
    Mar 2005
    Posts
    231
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I should note that mysql is hosted on windows.

  3. #3
    SQL Consultant gold trophysilver trophybronze trophy
    r937's Avatar
    Join Date
    Jul 2002
    Location
    Toronto, Canada
    Posts
    39,260
    Mentioned
    60 Post(s)
    Tagged
    3 Thread(s)
    Quote Originally Posted by bruin03 View Post
    What do you think caused this?
    you ran out of memory


    Quote Originally Posted by bruin03 View Post
    What other alternatives are out there for a database of this magnitude?
    break your csv up into chunks
    rudy.ca | @rudydotca
    Buy my SitePoint book: Simply SQL
    "giving out my real stuffs"

  4. #4
    SitePoint Addict
    Join Date
    Mar 2005
    Posts
    231
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by r937 View Post
    you ran out of memory


    break your csv up into chunks

    Good advice!

    I think that is the best practice moving forward to keep me from getting more gray hairs. I was ultimately able to do it using navicat.
    My system has an i7 9300processor and 6GB DDR3 RAM. I think Navicat handled the memory much more efficiently.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •