I. INTRODUCTION

“You can't manage what you don't measure.” This sentence is alleged to have been claimed by Edward Deming or Peter Drucker [1] and explains the importance of measurement. The Korea Ministry of National Defense (MND) is conducting evaluations in various fields to obtain meaningful effects from IT investment. The evaluation in the defense informatization sector is divided into defense informatization policy evaluation and defense informatization project evaluation [2], [3]. The defense informatization policy evaluation assesses various policies of MND and military forces related to the defense informatization, and the defense informatization project evaluation assesses IT projects such as information system (IS) development projects, IT procurement projects, and IS operation projects that are being carried out by the MND, Army, Navy, and Air Force [4]. On the other hand, the defense informatization level evaluation can measure the informatization level such as the informatization capacity of the MND and the related organizations or agencies.

Defense informatization aims to achieve military missions and create results by utilizing information technology. If the level of defense informatization associated with input or process factors is improved, the outcome factor, which is a performance or net effect, can be improved. The purpose of measuring the level of informatization can be improving the level of informatization (input or process factors) apart from the performance management of informatization (output factors).

This study proposes the evaluation system to measure the level of defense informatization in terms of defense information network, computer systems, interoperability and standardization, information security, informatization environment, and IS use, and also describes quantitative evaluation metrics in each field.

II. RELATED WORKS

2.1. Information Systems Success Model

DeLone and McLean [5] suggested an information systems (IS) success model (Fig. 1) after reviewing various existing researches related to IS performance. The model suggests that both IS quality and information quality affect both IS use and user satisfaction, that both IS use and user satisfaction affect the individual performance of IS, and that the individual performance lastly affects organizational performance. In addition, the IS use affects and is affected by the user satisfaction. In the view of the IS success model, the IS performance depends on IS quality and information quality, which is related to an input factor, IS use and user satisfaction, which is a process factor.

Figure. 1. DeLone and McLean’s IS success model

DeLone and McLean [6] reviewed the many empirical studies applied with their model [5], and suggested updated IS success model (Fig. 2). They included service quality and intention to use in their IS success model [5] and added a feedback from net benefits to the intention to IS use and the user satisfaction. In the view of the updated IS success model, the net benefits (performance) depends on IS quality, information quality, service quality, intention to use, use, and user satisfaction. This model focuses quality of systems, information, service, IS use, user satisfaction as an informatization level.

Figure. 2. DeLone and McLean’s updated IS success model.

2.2. ITU ICT Development Index

International Telecommunication Union (ITU) measures the level of informatization for each country by calculating the Information and Communication Technology Development Index (IDI) [7, 8, 9]. IDI is measured in Information and Communication Technology (ICT) access, ICT use, and ICT skills. ICT access measures the fixedtelephone subscriptions per 100 inhabitants, the mobilecellular telephone subscriptions per 100 inhabitants, the international Internet bandwidth (bit/s) per Internet user, the percentage of households with a computer, and the percentage of households with Internet access.

ICT use applies metrics such as the percentage of individuals using the Internet, the fixed-broadband subscriptions per 100 inhabitants, and the active mobilebroadband subscriptions per 100 inhabitants. ICT skills uses the metrics of the adult literacy rate, the secondary gross enrolment ratio, and the tertiary gross enrolment ratio.

The weights for ICT access, ICT use, and ICT skills are 40%, 40%, and 20%, respectively. The same weights are assigned to the metrics within each item. In 2017, ITU member countries decided to revise and expand the IDI [10], but the results have not yet been published as of October 2019. Infrastructure, use, and skills (user capability) are evaluated in the IDI.

2.3. Evaluation Method of Korean Government

The Korean government's evaluation of informatization capability (level) is being implemented as part of the evaluation of administrative management capability to improve the system in the informatization field [11]. This evaluation is managed by the Ministry of the Interior and Safety, which is based on the Framework Act on Public Service Evaluation [12]. Organizations to be evaluated are 44 central government departments including the MND.

Table 1. Metrics for national informatization level evaluation in Korea.

The Ministry of the Interior and Safety conducts a preliminary evaluation of the informatization field with the National Information Society Agency (NIA), and notifies the preliminary evaluation results to each central government department. The central administrative departments finally evaluate their informatization level themselves with the results. 

The evaluation system of the informatization field in 2019 shows that a perfect score is 35 points, and measures the implementation of effective e-government (20 points) and the reinforcement of cyber safety level (15 points). Evaluation metrics and their weights are shown in Table 1. The evaluation system also describes the evaluation formula as well as the evaluation method for each metric [11].

A survey on informatization level for small and medium enterprises (SMEs) is conducted by the Ministry of SMEs and Startups and Korea Technology & Information Promotion Agency for SMEs. This survey is for establishing informatization strategies and establishing policy directions for supporting SMEs [13]. It examines the general status of companies, the willingness and plans for informatization reinforcement, the environment for informatization reinforcement, the current status of IS implementation and use, the level of informatization effectiveness, and smart factories and new ICT technologies. Since 2012, the evaluation of SMEs informatization level has not been carried out using all the surveyed items, but only three items: the willingness and plans for informatization reinforcement, the environment for informatization reinforcement, and the current status of IS implementation and use.

Many researches have been conducted on the evaluation system for defense informatization level and the metrics for each item. Lim et al. [14] suggest to measure the level of defense informatization in terms of informatization infrastructure, informatization environment, informatization use, and informatization performance. In the informatization infrastructure, defense information network, computer systems, interoperability and standardization are measured. In the informatization environment, information security, organization, education, investment, and strategy of informatization are measured. The use of informatization measures the use of battlefield management information system (MIS) and resource MIS. The informatization performance measures the level of informatization combat power and the improvement of the efficiency of defense management. Also, various evaluation metrics are presented in the study [14]. The weights of evaluation items or metrics were calculated using Analytic Hierarchy Process (AHP). It is speculated that a method of calculating the evaluation metrics may have been developed, but the study [14] did not provide it. In order to demonstrate the usability of the developed evaluation system for defense informatization level, the informatization level was measured for military units in the operation command or higher with surveys on current status and questionnaires. For other methods of informatization level measurement, such as the corporate informatization level evaluation, and a survey on the SMEs informatization level prior to 2012, refer the existing study [14] 

Since the evaluation system of defense informatization level is mainly composed of qualitative evaluation metrics, it is necessary to reconstruct it based on quantitative evaluation metrics that can guarantee objectivity [15]. It should be able to manage the increase and decrease of the level by evaluation area. It should be designed with a focus on homeostasis of metrics so that it can be measured periodically. In addition, it is necessary to refrain from evaluating all evaluation metrics using the 5-point scale without a clear comparison, and to promote the evaluation in terms of achieved performance against targets. The method of measuring the informatization level with a questionnaire is simple, but there are limitations. This may be a subjective result by the survey respondent's perception rather than an objective one. The informatization level may be measured differently by the change of the survey respondent even though the organization or institution was the same informatization level. Longitudinal analysis also needs to be available for level improvement or trend analysis.

The level evaluation can be performed in the manner of improvement of metrics compared to the previous year, the level comparison to domestic or overseas organizations (institutes), and absolute maturity level. In the short term, it is advisable to conduct a level assessment in order to compare the level of metric improvement compared to the previous year or to compare the level of metrics with other organizations. To use the evaluation system for the informatization maturity level, at least several years of data must be accumulated and available.

III. EVALUATION FOR DEFENSE INFORMATIZATION LEVEL

This study describes an evaluation system for measuring the level of defense informatization. Defense informatization level assessment can be classified into IT infrastructure, informatization environment, and IS use according to the domain of evaluation (Table 2). In particular, the level assessment for IT infrastructure is evaluated by subdividing it into areas such as Defense information network (A), Computer systems (B), Interoperability and standardization (C), and Information security(D) [15]. In the Defense information network (A), speed, traffic, and availability/recovery time are measured. Server utilization, server availability, server throughput, and work efficiency are measured in Computer systems(B). Interoperability among ISs, standardization of reference information (master data), and interoperability between weapon systems and IS are measured in Interoperability and standardization (C). The detection level (capability) of infringement incident, the response level (capability) of infringement incident, and the recovery of infringement incident are measure in Information security (D).

Table 2. Evaluation metrics for defense informatization level.

Table 3. Metrics for Speed (A.1)

The level evaluation for defense information network is performed using metrics for Speed (A.1), Traffic (A.2), and Availability/recovery time (A.3). Table 3 shows the evaluation metrics for Speed (A.1). These are average speed among main nodes (main lines), average speed between the main node and the secondary node (branch line), and average speed between the secondary node and the user (user network). The equation is the ratio of measured speed of network against maximum speed (designed speed) to the target ratio. The data for network speed can be acquired from network management systems (NMS).

Table 4. Metrics for Traffic (A.2).

Table 4 describes metrics for traffic. The evaluation metrics for Traffic (A.2), like the metrics for Speed (A.1), measure a traffic on trunk, branch, and user network. These measure if an excessive traffic against a designed or estimated traffic occurs. The data for network traffic can be acquired from NMS.

The evaluation of Availability/recovery time (A.3) metric calculates availability with a probability that the network can be serviced without interruption, and manages data on network average recovery time to calculate availability. Table 5 presents a metric for network availability. The data for network availability can be acquired from NMS. The mean time to failure (MTTF) is average time of network service between failure of network and failure of network.

Table 5. Network availability metric.

IV. CONCLUSION

This study describes the defense informatization level evaluation system modified from the defense informatization evaluation methodology [4]. The proposed evaluation system for the defense informatization level measures in terms of IT infrastructure, informatization environment, and IS use [15]. This does not focus on a survey method but the direct measurement of the level if possible. The measurement effort should be put into the evaluation. The evaluation method that was being implemented by the government is accommodated as much as possible so that the efficient evaluation can be carried out while reducing the burden on the defense organization.

As with most researches and methodologies, there are limitations in the proposed evaluation system. It is necessary to set the target value for each metric. One also needsto consider metrics such as power usage effectiveness (PUE), which are used as metrics for US government data centers[16,17]. In addition, the study should have provided some cases or results applying the proposed evaluation system, but it does not due to a defense security issue.

Rather than waiting for the development of a sufficiently reasonable and theoretically complete evaluation system, it is better to complement the evaluation system by measuring the level of informatization. It is more significant to develop and apply the evaluation system that users can intuitively understand or use. Accumulated experiences with repetitive use of the evaluation system can lead to lessons learned and complementary needs, which can make the evaluation system more robust. Through such a virtuous cycle, the evaluation system of the defense informatization level and measurement metrics that are actively accepted by the stakeholders of the evaluation can be developed.

Acknowledgements

This manuscript is based on Research Report [15]. The authors wish to thank the editors and the anonymous reviewers for their careful reviews and constructive suggestions. Their suggestions helped strengthen the paper. All errors are the sole responsibility of the authors.