python mysql db migration script?

Discussion in 'Programming' started by CarpCharacin, Jun 25, 2016.

Thread Status:
Not open for further replies.
0
Page 1 of 2
  1. #1
    How would i make a python mysql database migration script? I want to migrate a forum mysql database forum a board running IPB 2.1 to XenForo. I posted in the XenForo subforum, but i did not get any help, so that is why i am posting here.
     
    CarpCharacin, Jun 25, 2016 IP
  2. Vooler

    Vooler Well-Known Member

    Messages:
    1,146
    Likes Received:
    64
    Best Answers:
    4
    Trophy Points:
    150
    #2
    First of all, the migration script may be written in any programming language, as you require mysql db be migrated and tuned for XenForo, any programmer willing to do that would need to understand data-structures of both database driven applications, i.e. Invison Power Board as well as XenForo.

    If you can post structure of tables of both scripts (only structure), some amongst programmers can give it a quick try. But it is a little time consuming.

    Stay well...
     
    Vooler, Jun 26, 2016 IP
  3. PoPSiCLe

    PoPSiCLe Illustrious Member

    Messages:
    4,623
    Likes Received:
    725
    Best Answers:
    152
    Trophy Points:
    470
    #3
    You will also have to understand the underlying code for the actual boards - depending on what you want to transfer. If you want to migrate users, posts, content, settings, roles/usertypes etc. you'll probably have to rewrite quite a bit of the data-structure, if the boards uses different ways of controlling stuff (which they probably do). Also remember that passwords aren't necessarily transferable, that depends entirely on how they're hashed - if the old hash-algorithm isn't compatible with the new algorithm, at best every user have to register a new password.
     
    PoPSiCLe, Jun 26, 2016 IP
    Vooler likes this.
  4. CarpCharacin

    CarpCharacin Active Member

    Messages:
    325
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    90
    #4
    I also want to migrate the media gallery.

    So do i post the structure like what is seen seen in phpmyadmin? Like a screenshot?
     
    Last edited by a moderator: Jun 27, 2016
    CarpCharacin, Jun 26, 2016 IP
  5. sarahk

    sarahk iTamer Staff

    Messages:
    28,816
    Likes Received:
    4,536
    Best Answers:
    123
    Trophy Points:
    665
    #5
    There will also be links within posts that need to be updated. Not a task for the fainthearted.

    I'd output the two file structures into .txt files, put them up on dropbox and post the links to them.

    As for the silence on the Xenforo forum - it's probably because nobody has done one of those migrations.
     
    sarahk, Jun 27, 2016 IP
  6. PoPSiCLe

    PoPSiCLe Illustrious Member

    Messages:
    4,623
    Likes Received:
    725
    Best Answers:
    152
    Trophy Points:
    470
    #6
    I doubt that one screenshot would do - you would need to have a complete list of which tables to migrate from, with all columns, and all the tables those tables are supposed to go into, with all columns. Also, you would need a list of non-matching columns, and some sort of list of what goes where, what you would have to provide yourself, and what, if anything, you will have to enter in manually. It's a big job, and you would probably need to set up a test-suite with a working IPB with content, and an empty XenForo setup so you can try and fail a couple hundred times.
     
    PoPSiCLe, Jun 27, 2016 IP
  7. sarahk

    sarahk iTamer Staff

    Messages:
    28,816
    Likes Received:
    4,536
    Best Answers:
    123
    Trophy Points:
    665
    #7
    This forum was converted from vBulletin which the Xenforo guys came from and probably were able to provide migration scripts. Even so it was a really, really long time between when the planning started to when the migration was rolled out. It was a side project but even so Shawn's no slouch and for someone relatively inexperienced and without any kind of framework to get you started you can expect to take twice as long.
     
    sarahk, Jun 27, 2016 IP
  8. CarpCharacin

    CarpCharacin Active Member

    Messages:
    325
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    90
    #8
    The site has roughly 32,000 members and 600,000 online posts. There are more archived posts so probably about 1 million posts. The site has been online since 1998. It has been on IPB for a long time. This one guy that i know is going to take it over from the current owner because it is not currently maintained. It just says that it could not determine the IP address when i go to it. The other site with 56,000 posts is going to get upgraded from phpbb 3.1 to xenforo and then i have to import the other site. Now i have the task of importing both sites. The phpbb one will be easy because there is a built in importer, but the IPB one will be hard. The IPB site had a high aquaranking when it was online, so it is worth importing.
     
    CarpCharacin, Jun 28, 2016 IP
  9. sarahk

    sarahk iTamer Staff

    Messages:
    28,816
    Likes Received:
    4,536
    Best Answers:
    123
    Trophy Points:
    665
    #9
    The size of the forum isn't really an issue. Catering for the complexity of the import process and testing cycles are what will take the time.
     
    sarahk, Jun 28, 2016 IP
    Vooler likes this.
  10. CarpCharacin

    CarpCharacin Active Member

    Messages:
    325
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    90
    #10
    Why does the import have to be so complex? Also, how will i redirect old URLs?
     
    CarpCharacin, Jun 28, 2016 IP
  11. sarahk

    sarahk iTamer Staff

    Messages:
    28,816
    Likes Received:
    4,536
    Best Answers:
    123
    Trophy Points:
    665
    #11
    Because there's a lot of information.

    Lets say this thread is in the IPB forum
    Here's a link: https://www.digitalpoint.com/members/carpcharacin.826280/
    and another one: https://forums.digitalpoint.com/threads/cost-of-setting-up-site-in-magento.2781445/

    So in this one post I have quoted text, click the little arrow and you get taken to the right post
    upload_2016-6-29_13-47-21.png
    Click either of those links and you expect to get to the right place.
    If you were just transferring it would be complicated enough
    but now I have to know that member 826280 is becoming member 65465461
    and post 19352209 is becoming post 654643216879543 and you have to update all the posts with the new number
    and thread 2781445 is becoming ...

    Your moderators are going to know what https://forums.digitalpoint.com/warnings/458832/ was about so you have to move that info across
    and there are all the private messages
    and likes
    and profile posts
    and moderator notes
    and in my account settings you need to know that what IPB calls X is called Y in Xenforo.

    and the list goes on
     
    sarahk, Jun 28, 2016 IP
  12. CarpCharacin

    CarpCharacin Active Member

    Messages:
    325
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    90
    #12
    How long will it take for me to do all of this? There is an importer for ipb 1.2. Could i use that? Also, what about avatars?
     
    CarpCharacin, Jun 28, 2016 IP
  13. sarahk

    sarahk iTamer Staff

    Messages:
    28,816
    Likes Received:
    4,536
    Best Answers:
    123
    Trophy Points:
    665
    #13
    I think you are starting to appreciate the scale of the job.
    I don't know what the differences are between your version of IPB and 1.2 - you'll have to review the upgrade docs, run a test or two.

    How long will it take? Depends on how detail oriented you are, how methodical, how quickly you learn but I'd be setting aside many, many evenings.
     
    Last edited: Jun 28, 2016
    sarahk, Jun 28, 2016 IP
  14. CarpCharacin

    CarpCharacin Active Member

    Messages:
    325
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    90
    #14
    It is IPB 2.1.7. I hope the site picks up activity again after the upgrade. It is a large site. At one time, it was one of the largest freshwater aquaria discussion sites online.
     
    CarpCharacin, Jun 28, 2016 IP
  15. sarahk

    sarahk iTamer Staff

    Messages:
    28,816
    Likes Received:
    4,536
    Best Answers:
    123
    Trophy Points:
    665
    #15
    The jump from v1 to v2 will be significant. I'd be ruling out that upgrade script being useful as anything more than a guide.
     
    sarahk, Jun 28, 2016 IP
  16. CarpCharacin

    CarpCharacin Active Member

    Messages:
    325
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    90
    #16
    IPB 1 and 2 look almost identical.
     
    CarpCharacin, Jun 28, 2016 IP
  17. sarahk

    sarahk iTamer Staff

    Messages:
    28,816
    Likes Received:
    4,536
    Best Answers:
    123
    Trophy Points:
    665
    #17
    lol, from whose perspective? the admin? the moderator? the user? the database administrator?
     
    sarahk, Jun 28, 2016 IP
  18. CarpCharacin

    CarpCharacin Active Member

    Messages:
    325
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    90
    #18
    I don't know about the admin or database admin, but from the user's perspective.
     
    CarpCharacin, Jun 28, 2016 IP
  19. sarahk

    sarahk iTamer Staff

    Messages:
    28,816
    Likes Received:
    4,536
    Best Answers:
    123
    Trophy Points:
    665
    #19
    A good site owner will try to achieve that so that users feel comfortable and secure - it's a bit like the duck: calm on the surface but paddling furiously. Just because a site or an upgrade looks simple/easy/insignificant don't be fooled that it is. Good usability and design might be making it look that way.

    upload_2016-6-29_17-8-16.png
     
    sarahk, Jun 28, 2016 IP
  20. PoPSiCLe

    PoPSiCLe Illustrious Member

    Messages:
    4,623
    Likes Received:
    725
    Best Answers:
    152
    Trophy Points:
    470
    #20
    Migrating one site platform to another is a daunting task, regardless of types of platforms involved. Unless there is an actual, existing and working migration script, you'll have quite a bit of work cut out for you.

    I suggest creating small test-cases based off of real data. First, find a single user (or create a new user) in the existing forum, let that user make some posts, get some likes, some quotes, some PMs, let it share some content, post images and links, and so on and so forth.
    Then you take everything about that user (first just the user, nothing the user has posted or made) and try to import that user into the new forum, to see what fails, what doesn't transfer over, and what you'll need to provide new information about. When THAT part is done, you start changing the original testuser's roles and states on the original forum, before you migrate him again. When EVERY POSSIBLE COMBO works, then you're ready for the hard part - migrating every member's posts and such. Given that you're starting with a freshly installed new forum, you can at least avoid having to change user ids and such (most likely), but you will probably have to update every single internal link there is - like @sarahk suggested in the previous post.

    It's gonna be a lot of work, and you will probably be fixing issues and bugs for a couple months afterwards.
     
    PoPSiCLe, Jun 29, 2016 IP
    sarahk likes this.
Page 1 of 2
Thread Status:
Not open for further replies.