Talk:Drupal/Archives/2013/June

CVE Table

So this user has been adding the CVE table back again. Reasons for the removal is here and the user did not motivate why those points were ignored.

The referenced data included modules/plugins for Drupal and Wordpress, perhaps the other CMSes too. This gives a very misleading picture of the actual security vulnerabilities as the modules/plugins are developed by other developers and not part of the actual CMS itself.

While quoting a source the author clearly misses the reductio ad absurdum of that source's methods: That the number of security issues reported is proportional to how insecure it is. If the number of security holes reported in an open source product shows how insecure it is then a product which reports no security problems must be the most secure.

217.211.59.24 (talk) 23:25, 28 May 2013 (UTC)

A Counterpoint View to the CVE Takedown

I would agree with a previous editor that CVEs are a valid measure to use. They may not be all inclusive; but, it is recognized by the industry and is non-biased. I’ll grant you, depending on use of a product, irrational conclusions could be drawn. If product A gets used say 1 billion times vs product B is only used 10 times and the conclusion was drawn that product A broke 1000 times while product B never broke, so B is better… that would be an ill comparison. However, this case is nowhere to that extreme. On the contrary, all the software being compared in the article is well used in the industry. While market shares are not dead even, their use is sufficient enough to derive valid statistical data from.

I would also like to comment that other industries use similar methodologies. Look at one of the larger industries of statistics… Auto insurance. A driver’s policy is largely in part based on statistical probability. A big factor in this is past performance… or more exactly… the failure to perform. It is a well-known fact that a driver who is involved in numerous accidents and moving violations is deemed a higher risk and flagged as an unsafe driver. The result of which is a higher premium paid by that driver. If the argument is that CVEs are irrelevant because it only shows a negative, then I can’t see how everyone sits idly by paying insurance premiums based off the same methodology.

The argument made with comparing the CVE and that one technology is better or worse (safe/unsafe) compared to another is just as valid as the insurance company saying Bob has 4 speeding tickets this past year while John has none… so Bob has to pay a higher premium because he’s deemed a less safe driver.

One aspect not addressed in the article would be CVEs in the previous 12 months or some similar metric. Something to say… “we weren’t that good in the past, but we are getting better” as opposed to just an “all time” score. Certainly, software that’s been around longer is likely to have more CVEs. Reaching back to the auto insurance, it would be like saying Bob who is 70 years old and driving for 50+ years has more speeding tickets than John who just got his license last year… so Bob is more of a risk because he has more speeding tickets. Insurance companies often look at short term performance knowing habits can change. So, perhaps a better or additional metric would be to offer a recent snapshot of accrued CVEs. I haven’t run those numbers… but my hunch is it won’t be anything in Drupal’s favor… which is also the primary naysayer in the validation of this method of comparison. For what it’s worth though, there were 4 software products reviewed. For comparison, the official release dates for the software are as follows: Drupal – 2001 (12 years), Wordpress – 2003 (10 years), Plone – 2003 (10 years), Joomla – 2005 (8 years). So while there is some disparity in total age, they are all well-established software.

To address a comment left previously by the unlogged and anonymized editor(s), “If we want the CVE information back we probably want to find better references that excludes [the plugins/modules].” For one, plugins are an integral part of the products being compared. Many plugins are even hosted on the software founders own web space. And, rarely are any of the products used without them. All vendors are being compared on even playing field. However, I would welcome a comparison without third-party contributions if one was available. The fact that a plugin is able to open a breach in the products security environment is almost as bad as the core product itself having the breach. If the plugin is the weak link in the chain, that would be an interesting metric to contrast alongside the core.

Having said that, I did a quick filter and comparison. As there are too many module related CVEs to determine which are core modules offered from the vendor and which are third party, all CVEs even mentioning the word “module” will be excluded for a quick comparison. Since the anon-editor is clearly pulling for Team Drupal, and the start of the CVE tables was spawned from the Plone Wiki, I’ll only contrast these two products.

The total list of lifetime CVEs

Technology	All time CVEs to date (2013-05-29)
Plone	21
Drupal	627

Now, filter the list for any CVE that contains the word “module” (again, this is excluding any module that may be provided by the vendor or related to how modules in general are handled) and the score comes down to this:

Lifetime CVEs excluding the word "module"

Technology	All time CVEs to date (2013-05-29)
Plone	19
Drupal	83

Still not fair? Filter it further to only look at 2012 & 2013 (the last calendar year and a half of CVEs) and it shows this:

Lifetime CVEs excluding the word "module"

Technology	CVEs from 2012-01-01 to 2013-05-29
Plone	0
Drupal	24

No, that isn’t a typo, there have been no CVEs egistered for Plone since 2011… and there are still 24 non-module related CVEs associated with Drupal in that same span of time. Does this imply that Plone is better? Does it vilify Drupal? That’s not for me to say. But from an auto insurance point of view, it would qualify Plone for the good driver discount. All this withstanding… all the compared software vendors are being judged and presented on an equal playing field. If there is a better metric to compare them, I’m open to seeing it used. In the interim, use what’s available.

In so far as the source of the metrics are concerned. I would like to point out that Mitre.org is a non-interested third party in the publication of the data.

“In partnership with government clients, The MITRE Corporation is a not-for-profit corporation working in the public interest. It addresses issues of critical national importance, combining systems engineering and information technology to develop innovative solutions that make a difference.”

Additionally, CVEs provide a basis for an unambiguous comparison.

“With CVE, your vulnerability databases, services, and tools can "speak" to each other. Until the creation of CVE in 1999 it was very difficult to effectively decide which tool was the most appropriate for an organization's needs. Each vendor used a different definition of "vulnerability" or "exposure" and used different metrics to state how many vulnerabilities or exposures they "check" or "test." CVE provides vendors with a standard list they can compare to, thus allowing you to compare apples to apples. In the longer term, CVE may be useful for obtaining quantitative data on tool behaviors, such as how they perform their checks, the impact they have on the systems they examine, the rate of false positives or false negatives, or how quickly they update their tools when new entries are introduced into CVE.”

Furthermore, the CVE list is considered reputable enough to be the feed for the U.S. National Vulnerability Database (NVD). If an additional reputable source is available, I would welcome its use. Any extra data for a side by side comparison would be welcomed. I expect that the numbers would tell the same story; but, if a comparison is available that says otherwise, I for one would be interested in seeing it. The anonymous editor is stating that these numbers are nonsensical and have no valid place in comparison. If so, explain what a “good comparison” is. Please explain what a “Proper measure of security” is. The previously included table simply showed that Plone has the lowest logged vulnerabilities. It offers numbers and links for the user to take into consideration for their own assessment and final judgment. Simply turning a blind eye to this data does not make it any less valid or important.

A quorum was asked for to discuss what people think about the data. As such.. I’m putting in my two cents. I ask that the previous editor step into the light, offer something constructive and stop de-posting valid data. Just removing the facts because you don’t like them doesn’t mean they aren’t true. If you don’t like the portrayal they give, find an alternative means to press your point of view rather than de-posting others contributions. As my two cents is more like a buck fifty due to its length… call this a flame if you wish. I’m simply laying all my cards out at once. I don’t want to get into a two sentence “I’m right/you’re wrong” editing war. As I believe this was the third time these have been removed, I will undo the previous takedown, make a few additional edits as I see fit (grammar issues and redundant statements abound anyway), and I will ask for the CVE portion of the pages to be locked until this is resolved.

Pursuant to Wikipedia’s guidance on non-consensus and deletion policy, I would ask that the anon-editor stop taking down the CVE table and instead offer a contribution.

-My apologies... I previously posted this comment to the archive instead of the live page. I followed an old link. I removed it from archive (why are people allowed to edit an archive anyway?) and moved it here where it should have been in the first place. chaydock (talk) 05:23, 31 May 2013 (UTC)