gamesover2600

[gamesover2600]

Migrating My ColdFusion Blog to DEV.to

I’ve decided to migrate my blog and dev-related posts to Dev.to.  My new home is https://dev.to/gamesover.

I currently plan on leaving existing posts here, but adding a message & link to the new dev.to entries. All new blog entries will be on the dev.to platform.

[gamesover2600]

StructIsOrdered() Support for ColdFusion 2016

Both ColdFusion 2016 and 2018 support ordered structs, but only ColdFusion 2018 has the built-in StructIsOrdered() function.  Here’s a UDF that adds fallback support for ColdFusion 2106.

NOTE: The StructIsOrdered() function doesn’t currently exist for Lucee.  I attempted to determine if the struct type was “linked” based on the canonicalName but the value was the same as an unordered struct. Interestingly enough, performing a CFDump displays “Struct (ordered)”.

https://gist.github.com/JamoCA/36784767c769bb8a11551ae481c5db73

[gamesover2600]

ColdFusion SetCookie UDF (Supports “SameSite”)

I attempted to post the following response regarding a ColdFusion bug that I reported in 2018, but Adobe’s CFTracker web application wasn’t working and refused to accept my post for an undisclosed reason. **

CFCookie "samesite" support** https://tracker.adobe.com/#/view/CF-4201688

A third-party site's API recently triggered the following warning message in Chrome 78:

A cookie associated with a cross-site resource at http://jetio.streamguys.com/ was set without the SameSite attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with SameSite=None and Secure. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032

I reported it to them and they fixed it within 2-3 days on their PHP platform.

As a workaround for CF2016, I'm using this modified UDF to set a CFCookie & a fallback "set-cookie" CFHeader.  It's duplicates the response headers, but if CFCookie isn't used, the value isn't added to the COOKIE scope.  (The 2nd set-cookie header overwrites the 1st one set by CF.) https://gist.github.com/JamoCA/bc34e27704eac277fcaf7053c54912b6

1/17/2020 Update

Adobe has indicated that this will be fixed in CF2016+, but it’s 20 days away and nothing has been made available yes.  On 1/16/2020, Google published Get Ready for New SameSite=None; Secure Cookie and listed other platforms that had same-site examples.

3/24/2020 Update

Adobe has posted manual patches for this on the bug report. 

(If using CF2016, download the CF2018 attachment as the instructions for CF2016 are incorrect.) If you are using ColdFusion 10 or 11, you’ll either have to use this UDF or Pete Freitag’s solution for IIS or Apache.

[gamesover2600]

My ColdFusion CFDocument Advice was “Marked as Spam”. Lame.

In response to: CFDocument Processing SpeedTest / Results and Inquiry https://community.adobe.com/t5/ColdFusion/CFDocument-Processing-SpeedTest-Results-and-Inquiry/td-p/10700365

We want to improve the performance of the CFDocument tag.  We are generating 6000 pdf pages using cfdocument by looping through a text file, formatting it and creating it using cfdocument tag. We are on CF 2018 Update 5.

Regardless of how much RAM and CPU we are throwing at CF, we cannot seem to convice CF that that RAM/CPU is eligible for use for this process. Whether the server has 16 GB RAM or 256 GB, the PMT shows that there is no differnce in the perfomance, and the amount of time it takes to finish processing doesn't seem to depend on either RAM or CPU. Should we be looking elsewhere (I/O) to speed this up, or is CFDocument simply the "little engine that could"?

We have been updating JVM heap sizes for minim and maximum and increased MaxMetaSpaceSize. At this time, there are no competing threads on this test server - just the CFDocument process.

Please let us know if there are any other settings or resources or other things to look for to increase the performance and generate the pdf faster. If not, we have found an alternative solution, but would prefer to learn more as to whether CFDocument processing can be improved. Thanks in advance.

I responded with advice that I both use & recommend as a ColdFusion Developer and it was immediately “marked as spam and removed.”  Here’s my response on 10/30/2019 06:45 Pacific.

As an alternative, you could save a single HTML to the file system and then try using the opensource WKHTMLTOPDF executable to generate the HTML.  (You can add page breaks & "keep sections together" using CSS.  A global header & footer requires 2 separate HTML templates.  "Page X of Y" can also be added using javascript.)  I shared a basic CFTag if you'd like to integrate within your project.  (Someone wrote a CFC, but I can't find it online anymore.)

We quit using CFDocument back in CF8.  Occasionally we'll compare the CFDocument output against WKHTMLTOPDF and we're still very glad that we're using WKHTMLTOPDF.  (I believe CFDocument uses outdated libraries and hasn't been updated in a very long time.)  WKHTMLTOPDF generates PDFs faster, requires less RAM, generates smaller PDF files and is compatible with CSS3/HTML5.  We've also got absolute-positioned scaled SVG images to work, whereas we weren't able to reliably generate 1px borders using CFDocument.  We were also able to integrate FontAwesome into our reports. Using WKHTMLTOPDF means that we can easily perform the HTML-to-PDF conversion in the background or separate server without using a single-threaded, long-timeout CFThread. (We use ajax or EventSource to determine completion status.)

10/31/2019 UPDATE: I noticed that my post was undeleted.  I didn’t receive any notification or alert, it just simply appeared and then I noticed that the original poster marked it as the answer. (Thanks Andy.)

[gamesover2600]

SQL Macros Using AutoHotKey (Windows)

Here are some AutoHotKey macros that I’ve written and use daily to significally reduce the amount of data transformations I have to perform with client-submitted data. The real benefit is that these macros work everywhere in Windows and aren’t isloated to function only in SQL Server Studio or VSCode... they’ll work anywhere (NotePad++, Outlook, Word, PowerShell, etc).  [NOTE: If you need to limit functionality to a specific app, it’s easy to add support so that these macros don’t overrule other programs.]

AutoHotKey script to prepends a prompted tablename variable to a list of SQL field names or variables from the clipboard. https://gist.github.com/JamoCA/94f3eae71c2d7cc52682f62c7cd3e984

AutoHotKey script to convert a column of clipboard data (copied from Excel, Word, Outlook, text file, etc) to a SQL list for use in a IN() query. https://gist.github.com/JamoCA/7737c8a18e1cfd7a6aecd79a19a96e03

[gamesover2600]

Identify SSL Expiration Date using ColdFusion

We use LetsEncrypt and monitor SSL certificate expiration dates internally by parsing the actual certificates, but if you require a way to remotely monitor a third-party SSL certificate in order to ensure that the renwal process didn’t fail, here’s some #cfml / #java to do just that.

https://gist.github.com/JamoCA/6aff53c0e002df8faa8c316f0138c507

[gamesover2600]

ColdFusion Dates (M/D/YYYYYYYYY)

While sanitizing user-provided data, I encountered a string that ColdFusion evaluated as a date, but SQL Server didn’t agree with due to “out of range” for the date SQL Server data type.  The date was “7/22/22019″.

I was curious to see what the max limit was and discovered that ColdFusion 2016 & 2018 behave differently when validating date. Apparently “12/31/292278993″ is a valid date in CF2016 & 2018, but the next year doesn’t exist in CF2016.  (“1/1 - 9/8″ month/days are valid in CF2016, but time ceases to exist on/after 9/10/292278993.)

I’m not sure why these dates are considered valid.  If I try to store these “dates” into either Microsoft SQL Server or MySQL, both will throw an “out of range” error. Even creating an inline java-based CFQuery throws an error. If you use CreateODBCDate() using ColdFusion 2018, it will throw a hard “uncatcheable” error which can’t be prevented using try/catch.

Since upgrading to ColdFusion 2016, I've had to write my own custom validation functions to deal with the shortcomings of Adobe's changes regarding how it validates integers, URLs, email... and now dates.

Here’s a quick proof-of-concept script: https://gist.github.com/JamoCA/01e14657a47846bb3aa7ca88295b8832

[gamesover2600]

Blocking HTML5 Ping Requests using ColdFusion

Major browsers are disabling the ability to disable HTML5 ping click tracking. https://www.bleepingcomputer.com/news/software/major-browsers-to-prevent-disabling-of-click-tracking-privacy-risk/

As a result, you’ll probably start encountering empty form posts with a content-type of “text/ping”.If you are not expecting or do not need to receive ping requests to your web server, you can block them without wasting any resources processing the request further.  This is important because this feature has already been used to perform DDOS attacks: https://www.imperva.com/blog/the-ping-is-the-thing-popular-html5-feature-used-to-trick-chinese-mobile-users-into-joining-latest-ddos-attack/

Here’s a basic ColdFusion script that will identify and block HTML5 Ping requests: https://gist.github.com/JamoCA/916dbb2d0ca0fe30ca63120bcaccc20f

[gamesover2600]

Unicode Domains Validated Differently Email vs URL

While sanitizing domains, I identified that ColdFusion (& Lucee) isValid() treats unicode domains differently when validating “email” versus “URL”. If java.netURL or HTML5 field URL validation is used to validate a URL containing a domain with a unicode character, it returns TRUE... but not isValid(”URL”).

I know that isValid(”email”) is currently too liberal and incorrectly returns TRUE if (“Testy McTest” <test@test.com>) is tested, but I’m not sure of the logic used for URL testing.  We’ve had issues in the past when isvalid() returned FALSE for Microsoft OneDrive URLs and stopped using built-in validation as we had no way of really knowing what works and what doesn’t.  (Does anyone have a publicly available unit test that Adobe uses for these functions?)

Here’s the Adobe Bug CF-4204516: https://tracker.adobe.com/#/view/CF-4204516

Here’s a GIST with sample domains: https://gist.github.com/JamoCA/4f9951753f2fe3ed79ef5d7945d2f926

Here’s a link to TryCF: https://www.trycf.com/gist/4f9951753f2fe3ed79ef5d7945d2f926

[gamesover2600]

Use Java replaceAll() instead of ColdFusion ReReplaceNoCase()

This blog post is in response to this blog post “Regular expressions – practical examples to get you started” https://coldfusion.adobe.com/2019/06/regular-expressions-practical-examples-get-started/ (My comment wasn’t accepted after submitting twice and I’d rather post it somewhere that I have a little control as Adobe’s community content tends to get deleted over time.)

We've encountered some unpreventable ColdFusion hard errors (that can't be prevented using try/catch) when using ReReplaceNoCase() in our application framework.  Here are some reported bugs that are similar: https://tracker.adobe.com/#/view/CF-3928688 https://tracker.adobe.com/#/view/CF-4165797

To work around this intermittently occuring bug (which Adobe classifies as "UserError" or "AsDesigned"), we started casting the initial value as a string (required for Java; not all CF-typeless variables contain pure "string" values) and using Java's replaceAll().  Our functions now seem faster, but more importantly there's no occassional CFError occurring in any of our apps.

When using regular expressions with ColdFusion, the ReEscape() function (available since CF10) simplifies escaping characters that match regular expression control characters. https://cfdocs.org/reescape

If interested in using more regex matching functions in ColdFusion, check out Ben Nadel's JRegEx: https://www.bennadel.com/blog/3322-jregex---a-coldfusion-wrapper-around-java-s-regular-expression-patterns.htm

[gamesover2600]

Spammers use “circled latin small letter g” to Bypass Spam Filters

I’ve been receiving some spam that has been difficult to detect/block and I’ve deteremined it’s because they use a high ASCII / UTF-8 character 9430 “circled latin small letter g” to bypass spam filters.

Using ColdFusion, I can send an email with a HTML link containing a domain with CHR(9430) as the “g” in “org”:

<a href="https://www.wikipedia.or#CHR(9430)#/">Click Me</a>

Interestingly enough, browsers convert this to a “g” and function as if everything is normal. If you copy the string and paste it into another program, it often is also converted to a regular lowercase “g”.

If I use Outlook to “save as MHTML”, then ".org" becomes " .or=E2=93=96″.  If I save as HTML, it becomes ". o r Ö$" and has strange spaces between the “o” and “r”.   This may be why why regular mail server filtering fails.  It’s either not seen as a valid domain or incapable of testing the actual intended “g” and performs a lookup on the extended, invalid character which may not ever be reported properly.

Anyone have recommendations on how to add filters to block this?

[gamesover2600]

Refresh Image with the Same Path without using cache-buster URL

Ever want to reload an image on a webpage that has a very long cache expiration date?  Most solutions that I’ve discovered and used in the past either added a hash to the image’s URL or changed the filename entirely.  I’ve written vanilla client-side javascript that functions in current versions Chrome & Firefox:

identifies the full URL file path of the image

identifies the width/height of the image and creates a temporary class so there’s no accordian affect during update

clears the image (using a data URI)

creates a hidden iFrame with an HTML fragment containing the image

forces a refresh on the iframe

updates the SRC tag of the original image

performs clean-up (removes temp classes & iframes)

Now I can upload new photos using ajax, use the exact same filename and have ithe image be immediately accessible (image refresh with updated cache.)  When comparing Chrome versus Firefox, this seems to function a lot faster when using Chrome.

https://gist.github.com/JamoCA/1233eba5a0ec38751eb666a0c72bb13c

[gamesover2600]

Multiple Forward Slashes in URL

While testing a permalink generator & manager library for ColdFusion, I came across a potential issue where the URL in the address bar didn’t exactly match the server-side permalink. In fact, I added 600 slashes to the URL and the webpage loaded without any issues.  I tried it on other websites and results varied a little bit. Microsoft IIS automatically normalizes the URI before passing it to ColdFusion as CGI.Path_Info.

I found a IIS Rewrite Rule online, but it didn’t fully work. I modified it so that it would only redirect GET requests and disabled "appendQueryString" so that the query string would be duplicated. This rule needs to be added as a local "per-site" rule instead of global since "matchType" isn't available at a global level. (I'm thinking about also disabling the rule for image, json & xml file requests.)

https://gist.github.com/JamoCA/5a873b93036c2baf0b5565abf1ac09f5

[gamesover2600]

Email Validation has Changed in ColdFusion 11, 2016 & 2018... or is it broken?

I recently upgraded an application to ColdFusion 2016 and clients started reporting issues with their outbound email sent through a third-party transaction email service.  We had a ColdFusion routine configured to check isValid(”email”) and, if invalid, parse the display name + email address so that we can pass the “name” and “email” separately.  This stopped working and started passing the entire “display name + email” as both the “name” and “email” as a result of an undocumented change.

This is extremely frustrating as no other technology I use will validate “display name + email” as a valid email address... not even the HTML5 INPUT field.

I’ve been burned by isvalid(”email”) before and wrote my own UDF that I’ve added my own rules around. (The UDF also uses a java isEmail library to perform an optional reverse DNS lookup for MX and checks to see that a server exists.)  I’ve updated our global UDF library to check for <, >, “ and spaces to determine if a “display name” is included. If detected, the string is immediately rejected as a valid email. This appears to be broken on ColdFusion 11, 2016, 2018 and all Railo/Lucee that I’ve used the string "Testy McTest” <test@test.com> with. (NOTE: You may want to change the test email to something else. My isEmail() UDF automatically rejects “test.com” because the domain is primarily used by bots, hackers & lazy marketing agents testing forms.)

If an email address consists of a “local-part, an @ symbol, then a case-insensitive domain“, why does CFML’s updated email validation also accept a “display name” value?  (Again, if you check the isValid() page, there’s absolutely no indication that the function behaves differently from how it has in the past.)  If you, or a third-party technology that you work with, doesn’t agree with Adobe, please upvote bug 4204100: https://tracker.adobe.com/#/view/CF-4204100

3/15/2019 UPDATE:  Email addresses over 73 characters will evaluate as “valid”, but CF2016u10 currently throws a “The label in the input is too long″ error when used in CFMail.  IP addresses are also accepted as domain names, but may not be routeable. Single word hostnames (no dot) are also valid (ie, “test@gmail”), but should not be used as DNS look-ups will likely fail. Check it out for yourself.  I have same sample CFML code here: https://gist.github.com/JamoCA/e160736354d1d9c21d181c0c0e988077

and an online demo here (I noticed that it sometimes throws an error even though the CFML is valid.): https://www.trycf.com/gist/e160736354d1d9c21d181c0c0e988077

Here’s a link to CFFiddle. https://cffiddle.org/app/file?filepath=186c5ed9-5a40-413b-bf42-4752506407fb/abe887b0-63a1-4720-8d4b-f1a3d95d219a/ddba9a15-8dd1-4c4c-bffb-7d8d0e60258d.cfm

[gamesover2600]

Convert Unicode strings to ASCII with ColdFusion & JUnidecode

I've struggled for years attempting to identify the best solution for converting unicode accents and other characters using ColdFusion.  I've used regex, java.text.Normalizer, ICU4J Transliterate and Apache.Lang3.StringUtils.StripAccents and recently scrapped them all in favor of using JUnidecode.  JUnidecode is a Java port of Text::Unidecode perl module.  The JUnidecode Java library only has one method and it takes a string and transliterates it to a valid 7-bit ASCII String (obviously it also strips diacritic marks).

Here's a demo script I've written that has some generic test cases: https://gist.github.com/JamoCA/6565bd4e2526b7c177a5f0cde3980d1c

[gamesover2600]

Local Proxy SVG Microicons using ColdFusion

I really like the idea of using a third-party for resources, but it's been painful relying on services over the years especially when they are retired with little or no annoucement.

There's a microservice called microicon that dynamically serves 1,200+ SVG icons from Material Design, Reline, Geomicons, and Simple Icons which allows the URL path to dictate size, color & direction.  I was interested in using this, but not if it was going to disappear sometime in the future and break all of my apps.  As a safeguard, I wrote a proxy script that will localize the remote SVG image in the background and serve them from the same host during future requests.

To accomplish this, SVG urls are required to match the IIS Rule and be prefixed with a "microicon" path and end with a "svg" file extension.  Here are the scripts that I developed.

https://gist.github.com/JamoCA/21520be121528a2b4ad361060c360a28

[gamesover2600]

Removing Empty Query Parameters from URL using ColdFusion

I recently read that ending website URLs with question marks diminishes its rank in the search algorithms.  After dumping a lot of server-side collections and reviewing variables, I discovered that ColdFusion 2016 has a CGI.HTTP_URL that is “sometimes” populated. I didn’t see this value ever populated when using ColdFusion 10 and determined it was indeed a bug (CF-2758070).  It was identified in CF11 and may (or may not) have been fixed based on the comments. (This isn’t the first time a bug has been reported and then only fixed in the next version.)  I wrote a function to identify GET page requests with empty query strings, remove the useless ending “?” character and then 301 redirect to the correct URL.  Enjoy.

NOTE: This script can’t be tested using TryCF.com or CFFiddle.org.

https://gist.github.com/JamoCA/eb0d47d0d179172e18991d95d853e359