Tool tModLoader: Browserfied (website version of mod browser)

Discussion in 'Released' started by SickBoy, Feb 3, 2018.

  1. SickBoy

    SickBoy Terrarian


    Hey there everyone! Long time listener, first time caller here... bringing you a web tool I made for myself because I just don't find the in-game tModLoader mod browser all that useful for certain actions, and because checking X number of threads every day just to see if an update occurred was getting tedious. I also figured it might be useful for those that can't actually use the mod browser in-game for whatever reason (eg: Linux/Mac users) and ease some of the load on the tModLoader server by minimizing access to the public mod list page. This is by no means meant to be a replacement for the mod browser, but rather an alternative for a niche audience.

    The nuts and bolts of how this works: my webserver pulls the publicly accessible PHP page for tModLoader mods every 12 hours (so as not to overload their server), splits all the data with a RegexFromHeckTM, compares mod version numbers with previously scraped data, updates everything, and dumps it all into a nicely formatted JSON file. Then I use some jQuery plugins to pop it all into a sortable, searchable grid. And bam, we now have a fully browser-accessible tModLoader mod database!

    Current Features:
    • All data is from the official tModLoader mod list, so *every* mod on the mod browser is available with the exact same data
    • Automatic data scraping of tModLoader's public PHP mod list page, so everything is (relatively) up to date
    • Search/sort by mod name
    • Search/sort by mod website/thread
    • Filter/sort by tModLoader version
    • Sort by update date (default view shows the newest first)
    • Manually download the .tmod directly from tModLoader servers
    • See the current version number
    • See the mod description used in the mod browser (click any row), now with color-code parsing REMOVED in 1.0.6 - tModLoader page no longer supplies it
    • Link to the mod website/thread
    • Seamless paging, etc. that you would expect from a tool of this kind
    • MUCH faster, easier, useful, and performance-friendly than tModLoader's public PHP mod list page
    • ZERO ads, trackers, CDNs, telemetry, analytics, Google/etc. scripts, social media stuff, account requirements, monetized links, etc... I don't want your info, and don't allow others to mine it either
    Preview: (this image is slightly outdated, but overall it's the same)
    preview.jpg
    Notes:
    • The PHP page I scrape has limited data available, and the only thing that I can use to flag a mod as being updated is the mod version number. I have never published a mod so I have no idea how that actually works in-game, but if it's possible to not update the version (accidentally or otherwise) then there's no way for my script to know. Everything *but* when it was updated will still be accurate though, as all the data is rewritten with every scrape.
    • If there is ever a change in the tModLoader PHP page structure, this tool will stop functioning. I don't expect it to anytime soon, but keep that in mind.
    • If a mod is removed from tModLoader, it will still show on the site until the next scrape occurs.
    • It should be obvious, but remember that this is not a truly realtime mod update list. You don't need to check it constantly, the absolute newest data will be absent for a few hours, and there's always a chance of temporarily dead download links if a mod is removed between scrapes.
    • If the in-game mod browser is down, the download links this tool supplies (and any attempts at data scraping) probably won't work either since they just link back directly to the tModLoader server, which redirects to the download location.
    • Tip: if you click a row, you'll get the mod description that is used in the mod browser. The More You Know! REMOVED in 1.0.6 - tModLoader page no longer supplies it
    Future Plans:
    • Let users bookmark specific mod entries to get a customized list to check?
    • Mod icon/most popular/etc. scraping maybe?
    • .tmod filename search?
    Changelog:
    Code:
    1.0.6 (2018-02-20)
    - Removed descriptions (no longer supplied by tModLoader page)
    - Rewrote regex (fix for removed descriptions)
    
    1.0.5 (2018-02-10)
    - Support for descriptions using the [c/ff0000:blah] coloration syntax
    - Blank descriptions given "No description added." text to prevent confusion
    - Link for changelog
    - Link for scrapelog
    - Link to discussion thread
    - Version text added
    - New default position for the dialog pop-up (top center of the grid)
    
    1.0.4 (2018-02-05)
    - Logging for scrape results (though not made public until 1.0.5)
    
    1.0.3 (2018-02-04)
    - Backend work to prevent certain problems if the tModLoader page can't be scraped
    - Backend security stuff
    - Changed "Last Updated:" date to AM/PM format
    - Fixed more "Last Updated:" date cases of oddness
    - Fixed a minor error in the math for determining how many days ago a mod update occurred
    
    1.0.2 (2018-02-04)
    - Text searching is now automatic (ie: after a slight delay while typing)
    - Fix for day display in "Last Updated:" date
    - Row selection prevented when clicking to download
    - Download column made easier to click
    - New confirmation+faq dialog box when clicking download links
    
    1.0.1 (2018-02-04)
    - New font
    - New backgrounds
    - New header banner
    - Tweaked footer banner
    - New styling for links
    - Larger text across the board
    - Dynamic resizing of the grid
    
    1.0.0 (2018-02-03)
    - Initial release
    
     
    Last edited: Feb 21, 2018
  2. SickBoy

    SickBoy Terrarian

    Some updates have occurred since the release yesterday. The tool is fancier in general, and some much-needed QoL changes were made. Everything seems to be working well (going live is always terrifying), but so far so good! I'm open to feedback and/or wishlist features to make this more useful and user-friendly, so let me know what you think... especially you Linux/Mac users out there, since this is mostly for your benefit.

    The changelog so far:
    Code:
    1.0.3 (2018-02-04)
    - Backend work to prevent certain problems if the tModLoader page can't be scraped
    - Backend security stuff
    - Changed "Last Updated:" date to AM/PM format
    - Fixed more "Last Updated:" date cases of oddness
    - Fixed a minor error in the math for determining how many days ago a mod update occurred
    
    1.0.2 (2018-02-04)
    - Text searching is now automatic (ie: after a slight delay while typing)
    - Fix for day display in "Last Updated:" date
    - Row selection prevented when clicking to download
    - Download column made easier to click
    - New confirmation+faq dialog box when clicking download links
    
    1.0.1 (2018-02-04)
    - New font
    - New backgrounds
    - New header banner
    - Tweaked footer banner
    - New styling for links
    - Larger text across the board
    - Dynamic resizing of the grid
     
    Last edited: Feb 5, 2018
  3. Battle Bee

    Battle Bee Terrarian

    Pretty nice! The in-game browser doesn't function for me, this is very helpful.
     
  4. SickBoy

    SickBoy Terrarian

    Thanks, glad to hear it! While the in-game browser works for me, I'm not a fan of its limitations, prefer to play offline (while still being able to check for updates manually), and can't stand trying to use the public PHP page the tModLoader folks supply... hence this tool. Figured others might find it helpful too, so I'm happy to know someone out there has a use for it. Here's to hoping they don't change that page's structure any time soon though... otherwise I'll be in a world of hurt to figure out the regex again. ;(
     
  5. SickBoy

    SickBoy Terrarian

    Finished another couple of updates in the past week. Nothing major, though the color tag parsing is nifty. Lack of updates is purely because most major changes were done pre-release, and there isn't much else to do until I get some feature requests.

    Code:
    1.0.5 (2018-02-10)
    - Support for descriptions using the [c/ff0000:blah] coloration syntax
    - Blank descriptions given "No description added." text to prevent confusion
    - Link for changelog
    - Link for scrapelog
    - Link to discussion thread
    - Version text added
    - New default position for the dialog pop-up (top center of the grid)
    
    1.0.4 (2018-02-05)
    - Logging for scrape results (though not made public until 1.0.5)
    Also, I looked into pulling download stats... and while all that info is available, the full list of total downloads doesn't have the same unique identifier (the download name, opposed to the mod name) that I use to guarantee the stats are applied properly. While the "downloads today/yesterday" list has that, the "downloads all-time" list doesn't. I did a quick once-over of the data, and there are a few that would indeed overlap if I just went off of the mod name. I could just do it anyways, since it's only a few, but it would add a huge amount of overhead to my processing script since I'd have to re-custom-process-and-combine the data yet again. So unless the tModLoader folks add the download name/link to the list, I can't really use it. A shame, but oh well.

    You may be asking why I don't just use the "downloads today/yesterday" data instead... well... it's just not nearly as interesting to me. I could just use that data to start keeping my own tallies in a similar manner as the update date, but it would probably get all kinds of broken due to not knowing when exactly their "yesterday" stats are generated. Something to consider, I suppose.

    Anyways, enjoy the update. As usual: let me know if there are any issues, requests, etc.
     
  6. Tyfyter

    Tyfyter Skeletron Prime

    "now I can finally update my first mod! Oh wait, I didn't make it open source..."
     
  7. SickBoy

    SickBoy Terrarian

    A little confused as to what you mean. Are you referring to your own train of thought, or suggesting I make this open source? If the latter: I can open it up if there is a desire, but I'm not entirely sure there would be a point to doing so. I also use a plugin (jqGrid) that I don't believe can be redistributed outside of the official site, since it's only free to use for non-commercial purposes. The only thing not exposed in the page source is my data scraping+processing script (I don't generate the user-facing page with anything server-side like PHP), which is a total mess... and verrrry specific to the tModLoader page that I scrape.

    As an example of the madness, here's the initial regex I use to split the data:

    regexmadness.jpg

    It could be trimmed down of course, but at a certain point I just wanted it to work... rather than be a gleaming example of regex godliness.
     
    Last edited: Feb 14, 2018
  8. Tyfyter

    Tyfyter Skeletron Prime

    I was just making a joke about the fact that I can't update my first mod because I no longer have the source code.
     
  9. Polyblank

    Polyblank Terrarian

    It’s been over 20 hours and the mod browser hasn’t updated yet?
     
    SickBoy likes this.
  10. SickBoy

    SickBoy Terrarian

    Thanks for letting me know! I had a feeling that happened... but I was so sleep-deprived that I couldn't figure out if the processor should have run or not (turns out converting 24 hour time format in your head is really confusing when you're super tired). Looks like the tModLoader folks changed the format of the public download listing page that I scrape. They no longer include the description of mods anymore. I'll go ahead and adjust my scripts accordingly.

    On one hand this is a good thing, because those descriptions took up something like 50-80% of the bandwidth of that (and my own) page, but it also means the usefulness of this tool just went down considerably... especially for the plethora of mods that don't have a homepage link and relied on the description for details on what it did. Maybe that'll encourage people to actually make a thread... :rolleyes:

    I'll see if I can contact someone on the tModLoader team to work out a solution that benefits everyone, but in the meantime I'll just chop out the description stuff as an emergency support effort.

    I'll report back when the update is complete.

    **EDIT**
    Done, manually ran the processor so everything should be up to date now. Saddened me to take all the description-related code out (especially since it was such a pain in the dingus to clean and process initially), but everything should be working now... just sans any descriptions. If the page doesn't look any different, refresh your browser cache with CTRL+F5 (or whatever your particular browser's equivalent is).

    Code:
    1.0.6 (2018-02-20)
    - Removed descriptions (no longer supplied by tModLoader page)
    - Rewrote regex (fix for removed descriptions)
     
    Last edited: Feb 21, 2018
    Polyblank likes this.
  11. iicecube

    iicecube Terrarian

    Not bad at least its much easy to get stuff done by this way..
    Neat and simple !
     
  12. SickBoy

    SickBoy Terrarian

    Thanks, that was definitely the intention. In my experience it's better to err on the side of simplicity when data organization is the goal, as it's all too easy to get sidetracked with fancy fluff that ultimately makes it harder to find what you want. A little bit of polish can go a long way, whereas too much just ruins the usability.
     
    iicecube likes this.
  13. Blackbluue

    Blackbluue Terrarian

    Really useful tool. Helps when my in-game mod browser is slow and/or unresponsive. My only suggestion is to indeed add user bookmarks so we can filter out only the mods we have installed without having to search up every single one individually. Gives it a bit of personalized touch similar to the in-game browser.
     
  14. Based Kamal

    Based Kamal Terrarian

    This is the greatest tool of all time.
    (to me at least, since mod browser is eternally offline)
     
  15. IDGCaptainRussia

    IDGCaptainRussia Skeletron

    Oh hey, this still works! It's a bit laggy thou but cool
    A bit surprised considering the author hasn't been on in almost a whole year
     
  16. Jofairden

    Jofairden Golem

    I wouldn't say that it works when it says:
    Last Updated: 2018/12/21 7:50pm Data is only processed once every 12 hours
     
  17. IDGCaptainRussia

    IDGCaptainRussia Skeletron

    slow, no doubt

    but he did say it was for a niche audience which does not include myself. So I don't go much else to criticize.
    Also might have misunderstood how it doesn't work, what you mean by that.
     
  18. Jofairden

    Jofairden Golem

    The data hasn't been updated in well over a month, when it says that it should update every 12 hours.
     
    IDGCaptainRussia likes this.
  19. IDGCaptainRussia

    IDGCaptainRussia Skeletron

    Well snap, Nothing can last without maintenance after all.
    Thanks for clarification.
     
  20. To be able to use the mods, are they instantly put into the game as playable mods, or do you need to put them into the game files like tmodloader?
    --- Double Post Merged, Jan 30, 2019, Original Post Date: Jan 30, 2019 ---
    Never mind, I figured it out!